Oct 30 19:30:47.438649 ip-10-0-142-167 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:30:47.872839 ip-10-0-142-167 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:30:47.872839 ip-10-0-142-167 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:30:47.872839 ip-10-0-142-167 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:30:47.872839 ip-10-0-142-167 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:30:47.881464 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881362    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:47.881464 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881460    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:47.881464 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881464    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:47.881464 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881469    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:47.881464 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881472    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:47.881464 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881475    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881478    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881481    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881483    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881486    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881489    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881491    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881494    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881496    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881499    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881501    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881504    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881506    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881509    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881511    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881514    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881517    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881519    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881522    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:47.881687 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881525    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881527    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881530    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881532    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881535    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881537    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881540    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881542    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881545    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881548    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881550    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881553    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881556    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881559    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881562    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881565    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881567    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881570    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881572    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881575    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:47.882124 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881578    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881580    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881583    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881585    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881588    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881591    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881595    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881598    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881601    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881603    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881606    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881609    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881611    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881614    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881617    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881619    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881622    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881625    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881628    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:47.882598 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881631    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881636    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881639    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881643    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881646    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881649    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881652    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881655    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881658    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881660    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881663    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881666    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881668    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881671    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881673    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881676    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881678    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881681    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881684    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:47.883039 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881686    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881691    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881693    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881696    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881699    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881702    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881704    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881707    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881709    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881711    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881715    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881717    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881720    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881723    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881725    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881729    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881733    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881736    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881739    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881742    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:47.883498 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881745    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881748    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881753    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.881755    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883781    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883789    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883793    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883795    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883798    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883801    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883803    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883806    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883809    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883811    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883815    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883817    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883820    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883823    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883825    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883828    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:47.883958 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883830    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883833    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883835    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883838    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883840    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883843    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883845    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883848    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883850    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883853    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883855    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883858    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883860    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883863    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883867    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883870    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883873    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883875    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883878    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:47.884550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883881    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883883    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883886    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883888    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883891    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883893    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883896    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883899    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883902    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883904    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883907    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883909    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883913    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883917    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883921    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883924    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883926    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883929    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883931    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:47.885005 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883934    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883936    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883939    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883941    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883943    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883946    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883948    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883951    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883954    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883957    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883960    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883963    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883965    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883968    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883970    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883973    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883975    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883978    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883981    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:47.885476 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883983    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883985    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883988    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883991    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883993    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883996    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.883998    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884000    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884003    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884005    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884008    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884010    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884013    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884015    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884017    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884020    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884022    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884025    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884027    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884030    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:47.885909 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884032    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884035    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884038    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884041    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884043    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884045    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884048    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884070    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884074    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884077    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884080    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884082    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884085    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884649    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884656    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884660    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884663    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884665    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884668    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884670    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:47.886411 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884673    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884676    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884678    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884681    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884683    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884686    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884689    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884691    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884694    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884697    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884700    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884702    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884705    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884708    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884711    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884713    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884716    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884718    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884721    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884723    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:47.886876 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884726    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884728    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884730    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884733    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884735    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884738    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884740    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884743    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884746    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884748    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884751    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884753    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884756    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884758    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884761    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884763    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884766    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884768    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884771    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:47.887375 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884773    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884776    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884778    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884781    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884784    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884786    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884789    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884792    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884794    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884796    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884799    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884801    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884804    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884806    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884809    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884811    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884813    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884816    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884819    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:47.887850 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884822    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884824    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884826    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884830    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884833    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884835    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884838    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884841    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884843    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884846    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884848    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884851    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884853    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884855    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884858    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884860    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884863    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884865    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884868    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884871    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:47.888281 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884873    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884876    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884878    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884881    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884883    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884886    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884888    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884891    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884894    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884896    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884898    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884901    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884903    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884906    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884910    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884914    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884919    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884923    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884926    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:47.888754 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884929    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.884932    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885039    2576 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885047    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885054    2576 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885059    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885064    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885067    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885074    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885078    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885081    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885084    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885087    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885091    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885094    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885097    2576 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885100    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885104    2576 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885108    2576 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885113    2576 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885119    2576 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885122    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885125    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885128    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:30:47.889218 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885132    2576 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885135    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885138    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885142    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885145    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885148    2576 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885152    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885155    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885158    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885161    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885165    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885168    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885171    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885174    2576 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885176    2576 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885180    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885185    2576 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885188    2576 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885191    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885194    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885197    2576 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885205    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885208    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885211    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885214    2576 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:30:47.889797 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885217    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885220    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885223    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885226    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885228    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885231    2576 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885234    2576 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885238    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885243    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885246    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885249    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885253    2576 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885256    2576 flags.go:64] FLAG: --help="false"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885259    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885263    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885266    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885269    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885272    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885275    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885278    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885281    2576 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885284    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885287    2576 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:30:47.890418 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885290    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885293    2576 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885296    2576 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885299    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885301    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885304    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885307    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885310    2576 flags.go:64] FLAG: --lock-file=""
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885313    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885316    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885319    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885325    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885327    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885330    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885333    2576 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885352    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885355    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885359    2576 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885362    2576 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885367    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885371    2576 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885375    2576 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885378    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885381    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885388    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:30:47.890950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885391    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885394    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885397    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885400    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885407    2576 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885410    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885413    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885416    2576 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885419    2576 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885422    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885425    2576 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885428    2576 flags.go:64] FLAG: --port="10250"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885430    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885433    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0886917c7464b7813"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885436    2576 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885440    2576 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885443    2576 flags.go:64] FLAG: --register-node="true"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885445    2576 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885449    2576 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885452    2576 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885455    2576 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885458    2576 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885461    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885464    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885468    2576 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885471    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:30:47.891565 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885474    2576 flags.go:64] FLAG: --runonce="false"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885478    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885481    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885484    2576 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885487    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885490    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885508    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885511    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885514    2576 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885517    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885520    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885523    2576 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885525    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885528    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885531    2576 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885535    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885540    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885543    2576 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885546    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885550    2576 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885552    2576 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885555    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885558    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885561    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885564    2576 flags.go:64] FLAG: --v="2"
Oct 30 19:30:47.892168 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885572    2576 flags.go:64] FLAG: --version="false"
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885576    2576 flags.go:64] FLAG: --vmodule=""
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885580    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.885583    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885728    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885733    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885737    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885739    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885743    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885746    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885748    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885750    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885753    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885755    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885759    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885761    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885764    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885766    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885769    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885771    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:47.892805 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885774    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885777    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885779    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885782    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885784    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885787    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885789    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885792    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885794    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885797    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885799    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885802    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885804    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885807    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885810    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885812    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885815    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885819    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885822    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:47.893271 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885826    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885834    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885838    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885841    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885843    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885846    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885848    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885851    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885854    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885857    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885859    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885861    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885864    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885867    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885870    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885872    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885875    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885878    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885880    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885883    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:47.893743 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885885    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885888    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885890    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885893    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885895    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885897    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885900    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885902    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885905    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885907    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885911    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885913    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885916    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885920    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885923    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885926    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885928    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885931    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885933    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885936    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:47.894210 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885939    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885942    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885944    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885947    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885949    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885952    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885954    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885957    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885959    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885961    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885964    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885966    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885969    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885971    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885974    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885976    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885979    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885981    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885984    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:47.894714 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885986    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:47.895164 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885989    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:47.895164 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885993    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:47.895164 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.885997    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:47.895164 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886000    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:47.895164 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886002    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:47.895164 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886006    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:47.895164 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886009    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:47.895164 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886011    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:47.895164 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886013    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:47.895164 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886016    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:47.895164 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886018    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:47.895164 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.886045    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:30:47.895164 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886085    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:47.895164 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886090    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:47.895164 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886094    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:47.895164 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886097    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886099    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886101    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886104    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886107    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886109    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886112    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886115    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886117    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886120    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886124    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886127    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886130    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886132    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886134    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886137    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886139    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886142    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886144    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886147    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:47.895657 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886150    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886152    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886155    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886158    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886160    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886163    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886165    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886168    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886170    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886173    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886175    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886178    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886181    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886183    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886186    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886188    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886190    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886193    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886195    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:47.896117 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886198    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886200    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886202    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886205    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886207    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886209    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886212    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886214    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886217    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886219    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886221    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886224    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886226    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886229    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886232    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886235    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886240    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886243    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886246    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:47.896582 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886248    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886251    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886253    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886256    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886258    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886261    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886263    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886266    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886268    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886271    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886273    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886275    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886278    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886280    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886283    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886285    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886288    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886290    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886292    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:47.897021 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886295    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886297    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886300    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886302    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886304    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886307    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886309    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886312    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886314    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886318    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886321    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886323    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886326    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886329    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886331    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886348    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886351    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886354    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886357    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886360    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:47.897507 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886362    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886365    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886368    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886370    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886372    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.886375    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.893656    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.893670    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.894459    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.894466    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894537    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894542    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894546    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894548    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894552    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894554    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894557    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894560    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894562    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:47.898016 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894565    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894567    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894571    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894573    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894576    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894579    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894582    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894584    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894587    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894589    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894592    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894594    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894597    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894599    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894608    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894611    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894613    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894616    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894620    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:47.898533 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894625    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894628    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894631    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894634    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894637    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894639    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894641    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894644    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894646    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894649    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894651    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894653    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894656    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894658    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894661    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894663    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894666    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894668    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894671    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894673    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:47.898981 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894676    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894678    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894681    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894684    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894686    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894689    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894692    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894694    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894702    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894705    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894707    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894710    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894713    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894715    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894718    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894720    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894723    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894726    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894728    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:47.899466 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894730    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894733    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894735    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894738    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894740    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894743    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894745    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894748    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894750    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894752    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894755    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894758    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894761    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894763    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894766    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894768    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894771    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894773    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894776    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894778    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:47.899908 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894781    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894783    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894791    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894794    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894796    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894799    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894801    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894803    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894806    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894808    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894811    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894813    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894815    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894818    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894820    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894822    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894825    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894827    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:47.900384 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.894829    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:47.900804 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.894858    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:30:47.900804 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895049    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:47.900804 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895055    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:47.900804 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895057    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:47.900804 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895062    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:47.900804 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895066    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:47.900804 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895068    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:47.900804 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895071    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:47.900804 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895074    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:47.900804 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895076    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:47.900804 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895078    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:47.900804 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895081    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:47.900804 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895083    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:47.900804 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895086    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:47.900804 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895088    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895091    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895099    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895102    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895104    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895107    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895109    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895111    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895114    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895116    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895119    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895121    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895123    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895126    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895128    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895130    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895133    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895135    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895138    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895140    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:47.901200 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895142    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895145    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895148    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895150    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895153    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895155    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895158    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895160    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895162    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895165    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895168    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895171    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895173    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895176    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895179    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895187    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895189    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895192    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895194    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895197    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:47.901682 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895199    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895201    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895204    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895206    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895208    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895211    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895213    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895216    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895218    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895221    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895223    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895226    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895228    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895231    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895234    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895236    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895239    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895241    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895244    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:47.902179 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895246    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895249    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895251    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895254    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895256    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895259    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895261    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895263    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895266    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895273    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895276    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895278    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895281    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895283    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895286    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895288    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895291    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895293    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895296    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:47.902692 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895300    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:47.903128 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895303    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:47.903128 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895305    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:47.903128 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895309    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:47.903128 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895311    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:47.903128 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895314    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:47.903128 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895316    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:47.903128 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895319    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:47.903128 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895321    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:47.903128 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895324    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:47.903128 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895326    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:47.903128 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895328    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:47.903128 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895331    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:47.903128 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895350    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:47.903128 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895354    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:47.903128 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.895379    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:30:47.903128 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895434    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:47.903128 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895438    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895441    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895444    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895446    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895449    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895451    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895460    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895462    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895465    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895467    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895469    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895472    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895474    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895476    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895479    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895481    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895484    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895486    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895488    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895491    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:47.903550 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895494    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895496    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895499    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895501    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895504    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895506    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895509    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895511    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895513    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895516    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895518    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895521    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895523    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895525    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895528    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895530    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895532    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895535    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895537    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895548    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:47.904010 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895551    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895553    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895556    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895558    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895561    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895563    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895565    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895568    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895570    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895573    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895575    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895577    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895580    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895583    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895586    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895588    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895590    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895593    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895595    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:47.904516 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895597    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895600    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895602    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895604    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895607    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895610    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895613    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895615    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895617    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895620    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895622    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895624    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895629    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895637    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895639    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895642    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895644    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895646    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895649    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:47.904956 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895651    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895655    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895659    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895662    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895665    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895668    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895671    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895673    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895676    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895678    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895681    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895683    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895686    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895688    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895691    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895693    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895696    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895698    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895701    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:47.905665 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895704    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:47.906433 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895707    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:47.906433 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895710    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:47.906433 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895717    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:47.906433 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895720    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:47.906433 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895722    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:47.906433 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895725    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:47.906433 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:47.895728    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:47.906433 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.895785    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:30:47.906433 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.899040    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 19:30:47.906433 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.900098    2576 server.go:1014] "Starting client certificate rotation"
Oct 30 19:30:47.906433 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.900236    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:30:47.906433 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.900277    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:30:47.906433 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.902498    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:30:47.907588 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.907568    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:30:47.924559 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.924535    2576 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:30:47.929206 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.929188    2576 log.go:25] "Validated CRI v1 image API"
Oct 30 19:30:47.930073 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.930047    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:30:47.933600 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.933580    2576 fs.go:135] Filesystem UUIDs: map[51d1ecf5-de09-4989-b83d-25c6833076d4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ae03064f-cef8-4f5f-a32d-09ae7eca4d98:/dev/nvme0n1p3]
Oct 30 19:30:47.933659 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.933600    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 19:30:47.939484 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.939324    2576 manager.go:217] Machine: {Timestamp:2026-10-30 19:30:47.937973312 +0000 UTC m=+0.382546070 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101000 MemoryCapacity:32811999232 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aa4968c9b4781fc841f5e476389cb SystemUUID:ec2aa496-8c9b-4781-fc84-1f5e476389cb BootID:360cc010-2d66-47c7-b98e-fe1974cfc051 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16405999616 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16405999616 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c1:a6:87:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c1:a6:87:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:9d:d5:69:ef:ea Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32811999232 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:30:47.939484 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.939477    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:30:47.939618 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.939570    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:30:47.939755 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.939738    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:30:47.940861 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.940832    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:30:47.941024 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.940864    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-167.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:30:47.941111 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.941030    2576 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:30:47.941111 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.941042    2576 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:30:47.941111 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.941088    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:30:47.941209 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.941113    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:30:47.942420 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.942407    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:30:47.946776 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.946761    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:30:47.946926 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.946915    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:30:47.946998 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.946963    2576 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:30:47.946998 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.946990    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:30:47.947083 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.947005    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:30:47.947083 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.947017    2576 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:30:47.947083 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.947050    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:30:47.947505 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.947491    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:30:47.950036 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.950000    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:30:47.957916 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.957899    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:30:47.958896 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.958883    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:30:47.958950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.958915    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:30:47.959080 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.959071    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:30:47.959114 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.959085    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:30:47.959114 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.959092    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:30:47.959114 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.959097    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:30:47.959114 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.959103    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:30:47.959114 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.959109    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:30:47.959973 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.959962    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:30:47.959973 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.959972    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:30:47.960055 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.959979    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:30:47.960055 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.959985    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:30:47.960055 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.959994    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:30:47.960055 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.960003    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:30:47.960769 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.960759    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:30:47.960769 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.960770    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:30:47.965631 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.965610    2576 server.go:1263] "Started kubelet"
Oct 30 19:30:47.965746 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.965710    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:30:47.966643 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.966590    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:30:47.966705 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.966669    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:30:47.967470 ip-10-0-142-167 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:30:47.968312 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.968299    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:30:47.969036 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.969022    2576 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:30:47.972405 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.972390    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-167.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 19:30:47.973359 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:47.973291    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 19:30:47.973700 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:47.973664    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-167.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 19:30:47.973776 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.973714    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mrgk9"
Oct 30 19:30:47.974256 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:47.974234    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 19:30:47.976865 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:47.975752    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-167.us-east-2.compute.internal.18e365793a306a45  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-167.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-167.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:30:47.965289029 +0000 UTC m=+0.409861791,LastTimestamp:2026-10-30 19:30:47.965289029 +0000 UTC m=+0.409861791,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-167.us-east-2.compute.internal,}"
Oct 30 19:30:47.979500 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.979482    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:30:47.979563 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.979501    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:30:47.979641 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:47.979622    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-167.us-east-2.compute.internal\" not found"
Oct 30 19:30:47.980257 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.980235    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:30:47.980257 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.980253    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:30:47.980434 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.980231    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:30:47.980434 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.980394    2576 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:30:47.980434 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.980401    2576 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:30:47.980726 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.980709    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:30:47.980726 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.980727    2576 factory.go:55] Registering systemd factory
Oct 30 19:30:47.980867 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.980736    2576 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:30:47.980953 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.980941    2576 factory.go:153] Registering CRI-O factory
Oct 30 19:30:47.980953 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.980954    2576 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:30:47.981046 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.980968    2576 factory.go:103] Registering Raw factory
Oct 30 19:30:47.981046 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.980981    2576 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:30:47.982061 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.982045    2576 manager.go:319] Starting recovery of all containers
Oct 30 19:30:47.985907 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.985881    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mrgk9"
Oct 30 19:30:47.992283 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.992266    2576 manager.go:324] Recovery completed
Oct 30 19:30:47.994441 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:47.994416    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 19:30:47.994441 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:47.994424    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-167.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 19:30:47.998905 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:47.998889    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:30:48.000658 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.000643    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:30:48.000718 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.000671    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:30:48.000718 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.000682    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:30:48.001157 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.001143    2576 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:30:48.001157 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.001156    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:30:48.001237 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.001171    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:30:48.001237 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.001203    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:30:48.001882 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.001866    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:30:48.001956 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.001894    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:30:48.001956 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.001904    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:30:48.003153 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.003140    2576 policy_none.go:50] "Start"
Oct 30 19:30:48.003194 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.003156    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:30:48.003194 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.003165    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:30:48.005082 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.005072    2576 policy_none.go:44] "Start"
Oct 30 19:30:48.044936 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.044908    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:30:48.047275 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.047258    2576 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:30:48.047352 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:48.047291    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:30:48.047352 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.047301    2576 server.go:87] "Starting device plugin registration server"
Oct 30 19:30:48.047576 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.047560    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:30:48.047673 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.047574    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:30:48.047723 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.047664    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:30:48.047774 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.047760    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:30:48.047816 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.047774    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:30:48.048242 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:48.048223    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:30:48.048319 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:48.048269    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-167.us-east-2.compute.internal\" not found"
Oct 30 19:30:48.136238 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.136162    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:30:48.138219 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.137557    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:30:48.138219 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.137586    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:30:48.138219 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.137611    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:30:48.138219 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:48.137656    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 19:30:48.142219 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.142196    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:30:48.148321 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.148296    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:30:48.149134 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.149119    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:30:48.149217 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.149147    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:30:48.149217 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.149157    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:30:48.149217 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.149183    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.158568 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.158552    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.158661 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:48.158578    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-167.us-east-2.compute.internal\": node \"ip-10-0-142-167.us-east-2.compute.internal\" not found"
Oct 30 19:30:48.174991 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:48.174964    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-167.us-east-2.compute.internal\" not found"
Oct 30 19:30:48.238726 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.238701    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal"]
Oct 30 19:30:48.238856 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.238780    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:30:48.239741 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.239724    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:30:48.239809 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.239764    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:30:48.239809 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.239783    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:30:48.241974 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.241962    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:30:48.242128 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.242112    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.242177 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.242143    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:30:48.242814 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.242799    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:30:48.242926 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.242826    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:30:48.242926 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.242841    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:30:48.242926 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.242803    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:30:48.242926 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.242893    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:30:48.242926 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.242904    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:30:48.244938 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.244919    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.245023 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.244952    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:30:48.245701 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.245684    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:30:48.245789 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.245720    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:30:48.245789 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.245737    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:30:48.269840 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:48.269819    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-167.us-east-2.compute.internal\" not found" node="ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.272351 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:48.272324    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-167.us-east-2.compute.internal\" not found" node="ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.275383 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:48.275370    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-167.us-east-2.compute.internal\" not found"
Oct 30 19:30:48.281533 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.281517    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1c62c9f273bf0b4293082033ce5e525-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"b1c62c9f273bf0b4293082033ce5e525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.281581 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.281547    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1c62c9f273bf0b4293082033ce5e525-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"b1c62c9f273bf0b4293082033ce5e525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.281581 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.281566    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2bdf808689747c42465325897cee67ca-config\") pod \"kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"2bdf808689747c42465325897cee67ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.376270 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:48.376233    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-167.us-east-2.compute.internal\" not found"
Oct 30 19:30:48.382568 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.382549    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1c62c9f273bf0b4293082033ce5e525-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"b1c62c9f273bf0b4293082033ce5e525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.382615 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.382577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1c62c9f273bf0b4293082033ce5e525-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"b1c62c9f273bf0b4293082033ce5e525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.382615 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.382595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2bdf808689747c42465325897cee67ca-config\") pod \"kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"2bdf808689747c42465325897cee67ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.382695 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.382668    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2bdf808689747c42465325897cee67ca-config\") pod \"kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"2bdf808689747c42465325897cee67ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.382695 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.382660    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1c62c9f273bf0b4293082033ce5e525-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"b1c62c9f273bf0b4293082033ce5e525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.382753 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.382668    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1c62c9f273bf0b4293082033ce5e525-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"b1c62c9f273bf0b4293082033ce5e525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.476782 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:48.476720    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-167.us-east-2.compute.internal\" not found"
Oct 30 19:30:48.572574 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.572542    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.575643 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.575618    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.577305 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:48.577160    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-167.us-east-2.compute.internal\" not found"
Oct 30 19:30:48.678125 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:48.678095    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-167.us-east-2.compute.internal\" not found"
Oct 30 19:30:48.778691 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:48.778633    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-167.us-east-2.compute.internal\" not found"
Oct 30 19:30:48.879176 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:48.879135    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-167.us-east-2.compute.internal\" not found"
Oct 30 19:30:48.899626 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.899594    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 19:30:48.899752 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.899735    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:30:48.963651 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.963627    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:30:48.979666 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.979638    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 19:30:48.979864 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.979852    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.993386 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.993314    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:25:49 +0000 UTC" deadline="2028-05-21 15:07:46.678272073 +0000 UTC"
Oct 30 19:30:48.993386 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.993379    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13651h36m57.68489631s"
Oct 30 19:30:48.995274 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.995259    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:30:48.996755 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.996741    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:30:48.998419 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:48.998403    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:30:49.005352 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:49.005324    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:30:49.070167 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:49.070097    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vg5wj"
Oct 30 19:30:49.082384 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:49.082362    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vg5wj"
Oct 30 19:30:49.144285 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:49.144261    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:30:49.267199 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:49.267149    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdf808689747c42465325897cee67ca.slice/crio-19eebbf5810a27b2a6adafe2b781d5d58f8a50f013fb9ac24422251ae6569f1e WatchSource:0}: Error finding container 19eebbf5810a27b2a6adafe2b781d5d58f8a50f013fb9ac24422251ae6569f1e: Status 404 returned error can't find the container with id 19eebbf5810a27b2a6adafe2b781d5d58f8a50f013fb9ac24422251ae6569f1e
Oct 30 19:30:49.267556 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:49.267533    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c62c9f273bf0b4293082033ce5e525.slice/crio-e6ccdc6ef5808392e98132ecb9055e6479ae1eab7bf7ad98ee84375a6adbb6a5 WatchSource:0}: Error finding container e6ccdc6ef5808392e98132ecb9055e6479ae1eab7bf7ad98ee84375a6adbb6a5: Status 404 returned error can't find the container with id e6ccdc6ef5808392e98132ecb9055e6479ae1eab7bf7ad98ee84375a6adbb6a5
Oct 30 19:30:49.271683 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:49.271664    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:30:49.390928 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:49.390857    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:30:51.352118 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.352007    2576 apiserver.go:52] "Watching apiserver"
Oct 30 19:30:51.363906 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.363867    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:30:51.364269 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.364248    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-7rv7g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp","openshift-cluster-node-tuning-operator/tuned-4pwl6","openshift-dns/node-resolver-dwfmh","openshift-image-registry/node-ca-bxxz2","openshift-multus/multus-vjc5d","kube-system/konnectivity-agent-qnmn2","openshift-multus/multus-additional-cni-plugins-64whk","openshift-multus/network-metrics-daemon-xmzrn","openshift-network-diagnostics/network-check-target-hr4sv","openshift-network-operator/iptables-alerter-45rm5","kube-system/global-pull-secret-syncer-6vm8t","kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"]
Oct 30 19:30:51.366939 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.366914    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.369267 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.369247    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.374167 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.374146    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.374278 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.374235    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:30:51.374886 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.374865    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:30:51.375014 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.374999    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g4c9f\""
Oct 30 19:30:51.375373 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.375359    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:30:51.375447 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.375417    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7mr7r\""
Oct 30 19:30:51.375821 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.375802    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:30:51.375918 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.375870    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:30:51.375918 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.375907    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:30:51.375918 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.375913    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:30:51.376081 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.375934    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:30:51.376557 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.376528    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:30:51.376646 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.376630    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:30:51.378184 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.378166    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5r9wt\""
Oct 30 19:30:51.378281 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.378200    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7c6bv\""
Oct 30 19:30:51.379067 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.378617    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:30:51.379067 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.378714    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:30:51.379067 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.378763    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:30:51.379067 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.378790    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:30:51.380158 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.380139    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:30:51.380245 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.380222    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.381751 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.381735    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qnmn2"
Oct 30 19:30:51.384655 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.384638    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.388770 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.387289    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:30:51.388770 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:51.387400    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:30:51.390861 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.390712    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:30:51.390861 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:51.390825    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:30:51.391826 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.391808    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:30:51.391919 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.391901    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vcqml\""
Oct 30 19:30:51.392063 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.391957    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-x9qfs\""
Oct 30 19:30:51.392063 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.392011    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ztltl\""
Oct 30 19:30:51.393242 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.393223    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:30:51.394381 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394304    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:30:51.394381 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394331    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:30:51.394531 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394382    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:30:51.394531 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394405    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:30:51.394531 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394409    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:30:51.394531 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394489    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:30:51.394531 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394492    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:30:51.394749 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394568    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:30:51.394749 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394583    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:30:51.394749 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394616    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:30:51.394749 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394696    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5aad5010-0827-4918-85d9-47e337a22774-ovn-node-metrics-cert\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.394749 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394726    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-cnibin\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.394975 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394751    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-socket-dir-parent\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.394975 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-slash\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.394975 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-var-lib-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.394975 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394893    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-run-ovn-kubernetes\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.394975 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-ovnkube-script-lib\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.395399 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.394977    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-cni-multus\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.395399 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395006    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-kubelet\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.395399 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395065    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fef01e5-9be4-4368-8f1e-3f73d7d6c60d-konnectivity-ca\") pod \"konnectivity-agent-qnmn2\" (UID: \"0fef01e5-9be4-4368-8f1e-3f73d7d6c60d\") " pod="kube-system/konnectivity-agent-qnmn2"
Oct 30 19:30:51.395399 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395107    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-systemd-units\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.395399 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395133    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-run-netns\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.395399 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395165    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-var-lib-kubelet\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.395399 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-tuned\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.395399 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-hostroot\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.395399 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395229    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.395399 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395287    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q94g\" (UniqueName: \"kubernetes.io/projected/6035c60e-c58e-455e-a61f-19eb599272af-kube-api-access-2q94g\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.395399 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395315    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb6r7\" (UniqueName: \"kubernetes.io/projected/ef93f664-f7a4-42b9-bb9c-a10c519e9685-kube-api-access-bb6r7\") pod \"node-resolver-dwfmh\" (UID: \"ef93f664-f7a4-42b9-bb9c-a10c519e9685\") " pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:30:51.395399 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395338    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bmxb\" (UniqueName: \"kubernetes.io/projected/b16dc00a-c3b0-4767-a468-6d204ba44e55-kube-api-access-7bmxb\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.395399 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395362    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-k8s-cni-cncf-io\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.395399 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395385    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-daemon-config\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.396071 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395413    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.396071 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395437    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcfwl\" (UniqueName: \"kubernetes.io/projected/ee3c157f-683e-415d-b856-74da8b4fe669-kube-api-access-kcfwl\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.396071 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs\") pod \"network-metrics-daemon-xmzrn\" (UID: \"a8eb8c11-ccd0-457f-92bd-c21db1c89e60\") " pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:30:51.396071 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-system-cni-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.396071 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395551    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-env-overrides\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.396071 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-cni-bin\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.396071 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395600    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-registration-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.396071 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-etc-selinux\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.396071 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-tuning-conf-dir\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.396071 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395670    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-systemd\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.396071 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395701    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.396071 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395729    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-ovnkube-config\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.396071 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395753    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-kubernetes\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.396071 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-device-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.396071 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395804    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fef01e5-9be4-4368-8f1e-3f73d7d6c60d-agent-certs\") pod \"konnectivity-agent-qnmn2\" (UID: \"0fef01e5-9be4-4368-8f1e-3f73d7d6c60d\") " pod="kube-system/konnectivity-agent-qnmn2"
Oct 30 19:30:51.396071 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395893    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b16dc00a-c3b0-4767-a468-6d204ba44e55-tmp\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.396813 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-cni-netd\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.396813 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysctl-conf\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.396813 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395973    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwlx\" (UniqueName: \"kubernetes.io/projected/35333d53-b864-432d-af6d-b1baec362af6-kube-api-access-mbwlx\") pod \"node-ca-bxxz2\" (UID: \"35333d53-b864-432d-af6d-b1baec362af6\") " pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:30:51.396813 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.395996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-cnibin\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.396813 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c2bb\" (UniqueName: \"kubernetes.io/projected/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-kube-api-access-7c2bb\") pod \"network-metrics-daemon-xmzrn\" (UID: \"a8eb8c11-ccd0-457f-92bd-c21db1c89e60\") " pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:30:51.396813 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-etc-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.396813 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396109    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84w8n\" (UniqueName: \"kubernetes.io/projected/5aad5010-0827-4918-85d9-47e337a22774-kube-api-access-84w8n\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.396813 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396133    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-lib-modules\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.396813 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-etc-kubernetes\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.396813 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396182    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef93f664-f7a4-42b9-bb9c-a10c519e9685-hosts-file\") pod \"node-resolver-dwfmh\" (UID: \"ef93f664-f7a4-42b9-bb9c-a10c519e9685\") " pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:30:51.396813 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396205    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-node-log\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.396813 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396229    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-modprobe-d\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.396813 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396250    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysconfig\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.396813 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396277    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-run\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.396813 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396299    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-host\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.396813 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396334    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dgf7\" (UniqueName: \"kubernetes.io/projected/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-kube-api-access-4dgf7\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.397466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396369    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-system-cni-dir\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.397466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396395    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.397466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396427    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-ovn\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.397466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396453    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysctl-d\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.397466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396475    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-sys\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.397466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396498    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef93f664-f7a4-42b9-bb9c-a10c519e9685-tmp-dir\") pod \"node-resolver-dwfmh\" (UID: \"ef93f664-f7a4-42b9-bb9c-a10c519e9685\") " pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:30:51.397466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396520    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-cni-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.397466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396544    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-cni-binary-copy\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.397466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-netns\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.397466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396716    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-socket-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.397466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396747    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-sys-fs\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.397466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-kubelet\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.397466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396815    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-log-socket\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.397466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396880    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-conf-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.397466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396943    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-multus-certs\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.397466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396980    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-cni-binary-copy\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.397953 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.396996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.397953 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.397016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-cni-bin\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.397953 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.397085    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-systemd\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.397953 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.397108    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:30:51.397953 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:51.397168    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:30:51.397953 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.397116    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35333d53-b864-432d-af6d-b1baec362af6-host\") pod \"node-ca-bxxz2\" (UID: \"35333d53-b864-432d-af6d-b1baec362af6\") " pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:30:51.397953 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.397208    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35333d53-b864-432d-af6d-b1baec362af6-serviceca\") pod \"node-ca-bxxz2\" (UID: \"35333d53-b864-432d-af6d-b1baec362af6\") " pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:30:51.398574 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.398546    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4mthb\""
Oct 30 19:30:51.472496 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.472457    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:30:51.472762 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.472735    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:30:51.472847 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.472835    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:30:51.472995 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.472960    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-79gww\""
Oct 30 19:30:51.485429 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.485407    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:30:51.488573 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.488522    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:25:50 +0000 UTC" deadline="2028-06-07 08:09:29.39424866 +0000 UTC"
Oct 30 19:30:51.488573 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.488572    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14052h38m37.905680464s"
Oct 30 19:30:51.498497 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.498465    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-registration-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.498612 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.498519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-etc-selinux\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.498612 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.498555    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-tuning-conf-dir\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.498713 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.498626    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-systemd\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.498762 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.498740    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-systemd\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.498908 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.498887    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.499080 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.499064    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-registration-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.499224 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.499209    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-etc-selinux\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.499409 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.499393    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-tuning-conf-dir\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.499470 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.499444    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-ovnkube-config\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.499523 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.499493    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-kubernetes\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.499569 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.499523    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-device-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.499569 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.499555    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fef01e5-9be4-4368-8f1e-3f73d7d6c60d-agent-certs\") pod \"konnectivity-agent-qnmn2\" (UID: \"0fef01e5-9be4-4368-8f1e-3f73d7d6c60d\") " pod="kube-system/konnectivity-agent-qnmn2"
Oct 30 19:30:51.499650 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.499582    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b16dc00a-c3b0-4767-a468-6d204ba44e55-tmp\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.499650 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.499623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xn78\" (UniqueName: \"kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78\") pod \"network-check-target-hr4sv\" (UID: \"be888c82-964b-42ae-948a-e6e27d276347\") " pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:30:51.499738 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.499676    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-cni-netd\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.499738 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.499706    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysctl-conf\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.499823 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.499763    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:30:51.499867 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.499834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwlx\" (UniqueName: \"kubernetes.io/projected/35333d53-b864-432d-af6d-b1baec362af6-kube-api-access-mbwlx\") pod \"node-ca-bxxz2\" (UID: \"35333d53-b864-432d-af6d-b1baec362af6\") " pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:30:51.499916 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.499858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-cnibin\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.499916 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.499898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2bb\" (UniqueName: \"kubernetes.io/projected/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-kube-api-access-7c2bb\") pod \"network-metrics-daemon-xmzrn\" (UID: \"a8eb8c11-ccd0-457f-92bd-c21db1c89e60\") " pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:30:51.500003 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.499926    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-etc-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.500003 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.499966    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84w8n\" (UniqueName: \"kubernetes.io/projected/5aad5010-0827-4918-85d9-47e337a22774-kube-api-access-84w8n\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.500112 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.500000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-lib-modules\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.500112 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.500029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-etc-kubernetes\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.500199 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.500173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef93f664-f7a4-42b9-bb9c-a10c519e9685-hosts-file\") pod \"node-resolver-dwfmh\" (UID: \"ef93f664-f7a4-42b9-bb9c-a10c519e9685\") " pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:30:51.500248 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.500196    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-cni-netd\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.500293 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.500265    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-node-log\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.500337 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.500300    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.500387 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.500349    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysctl-conf\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.500776 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.500741    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-cnibin\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.500850 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.500788    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-ovnkube-config\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.500900 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.500866    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-kubernetes\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.500944 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.500906    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-device-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.501061 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.501028    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-etc-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.501191 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.501173    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:30:51.501466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.501447    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-lib-modules\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.501527 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.501513    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-etc-kubernetes\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.501608 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.501579    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef93f664-f7a4-42b9-bb9c-a10c519e9685-hosts-file\") pod \"node-resolver-dwfmh\" (UID: \"ef93f664-f7a4-42b9-bb9c-a10c519e9685\") " pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:30:51.502103 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.500203    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-node-log\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.502183 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-modprobe-d\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.502228 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502194    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysconfig\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.502274 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502224    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-run\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.502274 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-host\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.502372 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7a73dcc-6a78-4344-b0b3-7a8611466a41-kubelet-config\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:30:51.502372 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502319    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysconfig\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.502466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502391    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-modprobe-d\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.502466 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dgf7\" (UniqueName: \"kubernetes.io/projected/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-kube-api-access-4dgf7\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.502560 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502458    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-system-cni-dir\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.502560 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502493    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.502560 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502499    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-run\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.502560 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-ovn\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.502560 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502557    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysctl-d\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.502801 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502566    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-host\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.502801 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502585    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-sys\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.502801 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502615    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef93f664-f7a4-42b9-bb9c-a10c519e9685-tmp-dir\") pod \"node-resolver-dwfmh\" (UID: \"ef93f664-f7a4-42b9-bb9c-a10c519e9685\") " pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:30:51.502801 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502649    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-cni-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.502801 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502692    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-cni-binary-copy\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.502801 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-netns\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.502801 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-socket-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.502801 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502780    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-sys-fs\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.503205 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502809    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-kubelet\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.503205 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-log-socket\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.503205 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502870    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-conf-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.503205 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502896    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-multus-certs\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.503205 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-cni-binary-copy\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.503205 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502958    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.503205 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.502992    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-cni-bin\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.503205 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503024    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-systemd\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.503205 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503064    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35333d53-b864-432d-af6d-b1baec362af6-host\") pod \"node-ca-bxxz2\" (UID: \"35333d53-b864-432d-af6d-b1baec362af6\") " pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:30:51.503205 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503095    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35333d53-b864-432d-af6d-b1baec362af6-serviceca\") pod \"node-ca-bxxz2\" (UID: \"35333d53-b864-432d-af6d-b1baec362af6\") " pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:30:51.503205 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503125    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5aad5010-0827-4918-85d9-47e337a22774-ovn-node-metrics-cert\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.503205 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503164    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-system-cni-dir\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.503205 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503164    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d432cc60-02df-4a8d-94d4-93aafac213f2-host-slash\") pod \"iptables-alerter-45rm5\" (UID: \"d432cc60-02df-4a8d-94d4-93aafac213f2\") " pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:30:51.503790 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503214    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.503790 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503231    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-kubelet\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.503790 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503284    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-log-socket\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.503790 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503298    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-cni-bin\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.503790 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503337    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35333d53-b864-432d-af6d-b1baec362af6-host\") pod \"node-ca-bxxz2\" (UID: \"35333d53-b864-432d-af6d-b1baec362af6\") " pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:30:51.503790 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503343    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-systemd\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.503790 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503357    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-multus-certs\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.503790 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503282    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-conf-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.503790 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503433    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-cnibin\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.503790 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503435    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-sys\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.503790 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503468    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-socket-dir-parent\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.503790 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503489    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-netns\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.503790 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503499    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-ovn\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.503790 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-slash\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.503790 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503616    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-socket-dir-parent\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.503790 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503731    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-cni-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.504526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.503883    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef93f664-f7a4-42b9-bb9c-a10c519e9685-tmp-dir\") pod \"node-resolver-dwfmh\" (UID: \"ef93f664-f7a4-42b9-bb9c-a10c519e9685\") " pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:30:51.504526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504023    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35333d53-b864-432d-af6d-b1baec362af6-serviceca\") pod \"node-ca-bxxz2\" (UID: \"35333d53-b864-432d-af6d-b1baec362af6\") " pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:30:51.504526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504153    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysctl-d\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.504526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504162    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-cni-binary-copy\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.504526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504202    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-slash\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.504526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504226    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-sys-fs\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.504526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504250    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-cnibin\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.504526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504262    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.504526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-var-lib-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.504526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504325    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-run-ovn-kubernetes\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.504526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504369    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-ovnkube-script-lib\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.504526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504394    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-cni-multus\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.504526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504477    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-kubelet\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.504526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504524    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fef01e5-9be4-4368-8f1e-3f73d7d6c60d-konnectivity-ca\") pod \"konnectivity-agent-qnmn2\" (UID: \"0fef01e5-9be4-4368-8f1e-3f73d7d6c60d\") " pod="kube-system/konnectivity-agent-qnmn2"
Oct 30 19:30:51.505211 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-systemd-units\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.505211 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-run-netns\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.505211 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504650    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-var-lib-kubelet\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.505211 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504674    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-tuned\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.505211 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504697    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7a73dcc-6a78-4344-b0b3-7a8611466a41-dbus\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:30:51.505211 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-hostroot\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.505211 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504759    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.505211 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504791    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q94g\" (UniqueName: \"kubernetes.io/projected/6035c60e-c58e-455e-a61f-19eb599272af-kube-api-access-2q94g\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.505211 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504786    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-cni-binary-copy\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.505211 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504831    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb6r7\" (UniqueName: \"kubernetes.io/projected/ef93f664-f7a4-42b9-bb9c-a10c519e9685-kube-api-access-bb6r7\") pod \"node-resolver-dwfmh\" (UID: \"ef93f664-f7a4-42b9-bb9c-a10c519e9685\") " pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:30:51.505211 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504873    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-var-lib-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.505211 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504934    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bmxb\" (UniqueName: \"kubernetes.io/projected/b16dc00a-c3b0-4767-a468-6d204ba44e55-kube-api-access-7bmxb\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.505211 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.504974    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d432cc60-02df-4a8d-94d4-93aafac213f2-iptables-alerter-script\") pod \"iptables-alerter-45rm5\" (UID: \"d432cc60-02df-4a8d-94d4-93aafac213f2\") " pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:30:51.505211 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.505008    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk9b4\" (UniqueName: \"kubernetes.io/projected/d432cc60-02df-4a8d-94d4-93aafac213f2-kube-api-access-lk9b4\") pod \"iptables-alerter-45rm5\" (UID: \"d432cc60-02df-4a8d-94d4-93aafac213f2\") " pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:30:51.505211 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.505062    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-k8s-cni-cncf-io\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.505211 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.505090    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-daemon-config\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.505211 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.505123    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.505981 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.505162    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcfwl\" (UniqueName: \"kubernetes.io/projected/ee3c157f-683e-415d-b856-74da8b4fe669-kube-api-access-kcfwl\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.505981 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.505194    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs\") pod \"network-metrics-daemon-xmzrn\" (UID: \"a8eb8c11-ccd0-457f-92bd-c21db1c89e60\") " pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:30:51.505981 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.505225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-system-cni-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.505981 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.505254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-env-overrides\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.505981 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.505286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-cni-bin\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.505981 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.505401    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-k8s-cni-cncf-io\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.505981 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.505427    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fef01e5-9be4-4368-8f1e-3f73d7d6c60d-konnectivity-ca\") pod \"konnectivity-agent-qnmn2\" (UID: \"0fef01e5-9be4-4368-8f1e-3f73d7d6c60d\") " pod="kube-system/konnectivity-agent-qnmn2"
Oct 30 19:30:51.505981 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.505477    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-systemd-units\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.505981 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.505537    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-var-lib-kubelet\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.505981 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.505666    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-run-netns\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.505981 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.505719    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-cni-multus\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.505981 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.505762    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-run-ovn-kubernetes\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.505981 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.505921    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-daemon-config\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.506433 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.506095    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-hostroot\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.507630 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.506750    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-cni-bin\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.507630 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:51.506779    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:30:51.507630 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.506818    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.507630 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.506827    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-kubelet\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.507630 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.506872    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-system-cni-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.507630 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.506873    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-ovnkube-script-lib\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.507630 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:51.506910    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs podName:a8eb8c11-ccd0-457f-92bd-c21db1c89e60 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:52.006883052 +0000 UTC m=+3.047749813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs") pod "network-metrics-daemon-xmzrn" (UID: "a8eb8c11-ccd0-457f-92bd-c21db1c89e60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:30:51.507630 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.507200    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-socket-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.507630 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.507326    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.507630 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.507346    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-env-overrides\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.507630 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.507414    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5aad5010-0827-4918-85d9-47e337a22774-ovn-node-metrics-cert\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.508439 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.507729    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fef01e5-9be4-4368-8f1e-3f73d7d6c60d-agent-certs\") pod \"konnectivity-agent-qnmn2\" (UID: \"0fef01e5-9be4-4368-8f1e-3f73d7d6c60d\") " pod="kube-system/konnectivity-agent-qnmn2"
Oct 30 19:30:51.508439 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.508205    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-tuned\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.508568 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.508544    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b16dc00a-c3b0-4767-a468-6d204ba44e55-tmp\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.515884 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.515814    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c2bb\" (UniqueName: \"kubernetes.io/projected/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-kube-api-access-7c2bb\") pod \"network-metrics-daemon-xmzrn\" (UID: \"a8eb8c11-ccd0-457f-92bd-c21db1c89e60\") " pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:30:51.516074 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.516015    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwlx\" (UniqueName: \"kubernetes.io/projected/35333d53-b864-432d-af6d-b1baec362af6-kube-api-access-mbwlx\") pod \"node-ca-bxxz2\" (UID: \"35333d53-b864-432d-af6d-b1baec362af6\") " pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:30:51.516164 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.516140    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dgf7\" (UniqueName: \"kubernetes.io/projected/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-kube-api-access-4dgf7\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.517908 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.517857    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb6r7\" (UniqueName: \"kubernetes.io/projected/ef93f664-f7a4-42b9-bb9c-a10c519e9685-kube-api-access-bb6r7\") pod \"node-resolver-dwfmh\" (UID: \"ef93f664-f7a4-42b9-bb9c-a10c519e9685\") " pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:30:51.518481 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.518457    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bmxb\" (UniqueName: \"kubernetes.io/projected/b16dc00a-c3b0-4767-a468-6d204ba44e55-kube-api-access-7bmxb\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.518589 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.518555    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84w8n\" (UniqueName: \"kubernetes.io/projected/5aad5010-0827-4918-85d9-47e337a22774-kube-api-access-84w8n\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.520464 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.520091    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcfwl\" (UniqueName: \"kubernetes.io/projected/ee3c157f-683e-415d-b856-74da8b4fe669-kube-api-access-kcfwl\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.522003 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.521982    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q94g\" (UniqueName: \"kubernetes.io/projected/6035c60e-c58e-455e-a61f-19eb599272af-kube-api-access-2q94g\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.547883 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.547829    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal" event={"ID":"2bdf808689747c42465325897cee67ca","Type":"ContainerStarted","Data":"19eebbf5810a27b2a6adafe2b781d5d58f8a50f013fb9ac24422251ae6569f1e"}
Oct 30 19:30:51.549265 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.549235    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal" event={"ID":"b1c62c9f273bf0b4293082033ce5e525","Type":"ContainerStarted","Data":"e6ccdc6ef5808392e98132ecb9055e6479ae1eab7bf7ad98ee84375a6adbb6a5"}
Oct 30 19:30:51.605930 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.605621    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xn78\" (UniqueName: \"kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78\") pod \"network-check-target-hr4sv\" (UID: \"be888c82-964b-42ae-948a-e6e27d276347\") " pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:30:51.605930 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.605667    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:30:51.605930 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.605700    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7a73dcc-6a78-4344-b0b3-7a8611466a41-kubelet-config\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:30:51.605930 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.605735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d432cc60-02df-4a8d-94d4-93aafac213f2-host-slash\") pod \"iptables-alerter-45rm5\" (UID: \"d432cc60-02df-4a8d-94d4-93aafac213f2\") " pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:30:51.605930 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.605797    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d432cc60-02df-4a8d-94d4-93aafac213f2-host-slash\") pod \"iptables-alerter-45rm5\" (UID: \"d432cc60-02df-4a8d-94d4-93aafac213f2\") " pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:30:51.605930 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:51.605824    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:30:51.605930 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.605852    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7a73dcc-6a78-4344-b0b3-7a8611466a41-kubelet-config\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:30:51.605930 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:51.605890    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret podName:b7a73dcc-6a78-4344-b0b3-7a8611466a41 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:52.10586948 +0000 UTC m=+3.146736238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret") pod "global-pull-secret-syncer-6vm8t" (UID: "b7a73dcc-6a78-4344-b0b3-7a8611466a41") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:30:51.606429 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.605954    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7a73dcc-6a78-4344-b0b3-7a8611466a41-dbus\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:30:51.606429 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.605986    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d432cc60-02df-4a8d-94d4-93aafac213f2-iptables-alerter-script\") pod \"iptables-alerter-45rm5\" (UID: \"d432cc60-02df-4a8d-94d4-93aafac213f2\") " pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:30:51.606429 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.606011    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk9b4\" (UniqueName: \"kubernetes.io/projected/d432cc60-02df-4a8d-94d4-93aafac213f2-kube-api-access-lk9b4\") pod \"iptables-alerter-45rm5\" (UID: \"d432cc60-02df-4a8d-94d4-93aafac213f2\") " pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:30:51.606429 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.606121    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7a73dcc-6a78-4344-b0b3-7a8611466a41-dbus\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:30:51.606751 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.606731    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d432cc60-02df-4a8d-94d4-93aafac213f2-iptables-alerter-script\") pod \"iptables-alerter-45rm5\" (UID: \"d432cc60-02df-4a8d-94d4-93aafac213f2\") " pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:30:51.615322 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:51.615292    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:30:51.615322 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:51.615323    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:30:51.615533 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:51.615337    2576 projected.go:196] Error preparing data for projected volume kube-api-access-5xn78 for pod openshift-network-diagnostics/network-check-target-hr4sv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:30:51.615533 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:51.615397    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78 podName:be888c82-964b-42ae-948a-e6e27d276347 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:52.11537961 +0000 UTC m=+3.156246371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5xn78" (UniqueName: "kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78") pod "network-check-target-hr4sv" (UID: "be888c82-964b-42ae-948a-e6e27d276347") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:30:51.618347 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.618320    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk9b4\" (UniqueName: \"kubernetes.io/projected/d432cc60-02df-4a8d-94d4-93aafac213f2-kube-api-access-lk9b4\") pod \"iptables-alerter-45rm5\" (UID: \"d432cc60-02df-4a8d-94d4-93aafac213f2\") " pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:30:51.701795 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.701751    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:30:51.706160 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.706130    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:30:51.712269 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:51.712235    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16dc00a_c3b0_4767_a468_6d204ba44e55.slice/crio-85506832c693909051574640612a0a3de3e99ac27535720e9b9e5d5fcbd87580 WatchSource:0}: Error finding container 85506832c693909051574640612a0a3de3e99ac27535720e9b9e5d5fcbd87580: Status 404 returned error can't find the container with id 85506832c693909051574640612a0a3de3e99ac27535720e9b9e5d5fcbd87580
Oct 30 19:30:51.713957 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.713938    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:30:51.714579 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.714513    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:30:51.717112 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:51.717067    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aad5010_0827_4918_85d9_47e337a22774.slice/crio-3ea2b374f029306389eb06c1bedfa13941608fc7e0fe310460a8d3b094709229 WatchSource:0}: Error finding container 3ea2b374f029306389eb06c1bedfa13941608fc7e0fe310460a8d3b094709229: Status 404 returned error can't find the container with id 3ea2b374f029306389eb06c1bedfa13941608fc7e0fe310460a8d3b094709229
Oct 30 19:30:51.723809 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.723783    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:30:51.725924 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:51.725900    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6035c60e_c58e_455e_a61f_19eb599272af.slice/crio-f59a3d2fa1f4bdde3a8fe04e2021f2bfc6b47b7e33ecd49e248c1665a784d27b WatchSource:0}: Error finding container f59a3d2fa1f4bdde3a8fe04e2021f2bfc6b47b7e33ecd49e248c1665a784d27b: Status 404 returned error can't find the container with id f59a3d2fa1f4bdde3a8fe04e2021f2bfc6b47b7e33ecd49e248c1665a784d27b
Oct 30 19:30:51.729470 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:51.729132    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef93f664_f7a4_42b9_bb9c_a10c519e9685.slice/crio-5a1a321b62a9b9de7d0b59b69e9430c2a74a4f6b70b2d28b081282567e2300b4 WatchSource:0}: Error finding container 5a1a321b62a9b9de7d0b59b69e9430c2a74a4f6b70b2d28b081282567e2300b4: Status 404 returned error can't find the container with id 5a1a321b62a9b9de7d0b59b69e9430c2a74a4f6b70b2d28b081282567e2300b4
Oct 30 19:30:51.729470 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.729237    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vjc5d"
Oct 30 19:30:51.736293 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.736273    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qnmn2"
Oct 30 19:30:51.737475 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:51.737446    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35333d53_b864_432d_af6d_b1baec362af6.slice/crio-fe2bb906042c92e3f81991602e039cca05ede3fc631204df00ceba6e679577b5 WatchSource:0}: Error finding container fe2bb906042c92e3f81991602e039cca05ede3fc631204df00ceba6e679577b5: Status 404 returned error can't find the container with id fe2bb906042c92e3f81991602e039cca05ede3fc631204df00ceba6e679577b5
Oct 30 19:30:51.740944 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:51.740910    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfe62e2_9594_4c48_a3bc_2dce9df78b56.slice/crio-55ec1086b671b271d3dfa310dedfc2c6b40729d72cba4ca9141a1a58756155cd WatchSource:0}: Error finding container 55ec1086b671b271d3dfa310dedfc2c6b40729d72cba4ca9141a1a58756155cd: Status 404 returned error can't find the container with id 55ec1086b671b271d3dfa310dedfc2c6b40729d72cba4ca9141a1a58756155cd
Oct 30 19:30:51.747337 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:51.747309    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fef01e5_9be4_4368_8f1e_3f73d7d6c60d.slice/crio-3b5fcf354f1de5ae721c7e59bdbc57b009d0fcd8dffc411c85af9784f33c2318 WatchSource:0}: Error finding container 3b5fcf354f1de5ae721c7e59bdbc57b009d0fcd8dffc411c85af9784f33c2318: Status 404 returned error can't find the container with id 3b5fcf354f1de5ae721c7e59bdbc57b009d0fcd8dffc411c85af9784f33c2318
Oct 30 19:30:51.755152 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.755127    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:30:51.760221 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.760196    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:30:51.767156 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:30:51.766996    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd432cc60_02df_4a8d_94d4_93aafac213f2.slice/crio-deae8580d1249abde68c8681ce29ed814663d86a100c519e5bc3ca68198b88ed WatchSource:0}: Error finding container deae8580d1249abde68c8681ce29ed814663d86a100c519e5bc3ca68198b88ed: Status 404 returned error can't find the container with id deae8580d1249abde68c8681ce29ed814663d86a100c519e5bc3ca68198b88ed
Oct 30 19:30:51.768516 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:51.768476    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:30:52.009513 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:52.009441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs\") pod \"network-metrics-daemon-xmzrn\" (UID: \"a8eb8c11-ccd0-457f-92bd-c21db1c89e60\") " pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:30:52.009672 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:52.009612    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:30:52.009732 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:52.009694    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs podName:a8eb8c11-ccd0-457f-92bd-c21db1c89e60 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:53.009672132 +0000 UTC m=+4.050538889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs") pod "network-metrics-daemon-xmzrn" (UID: "a8eb8c11-ccd0-457f-92bd-c21db1c89e60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:30:52.110545 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:52.110512    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:30:52.110698 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:52.110653    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:30:52.110752 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:52.110744    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret podName:b7a73dcc-6a78-4344-b0b3-7a8611466a41 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:53.110729854 +0000 UTC m=+4.151596596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret") pod "global-pull-secret-syncer-6vm8t" (UID: "b7a73dcc-6a78-4344-b0b3-7a8611466a41") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:30:52.211898 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:52.211858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xn78\" (UniqueName: \"kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78\") pod \"network-check-target-hr4sv\" (UID: \"be888c82-964b-42ae-948a-e6e27d276347\") " pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:30:52.212103 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:52.212081    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:30:52.212103 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:52.212105    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:30:52.212312 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:52.212116    2576 projected.go:196] Error preparing data for projected volume kube-api-access-5xn78 for pod openshift-network-diagnostics/network-check-target-hr4sv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:30:52.212312 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:52.212236    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78 podName:be888c82-964b-42ae-948a-e6e27d276347 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:53.212216344 +0000 UTC m=+4.253083099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5xn78" (UniqueName: "kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78") pod "network-check-target-hr4sv" (UID: "be888c82-964b-42ae-948a-e6e27d276347") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:30:52.489419 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:52.489343    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:25:50 +0000 UTC" deadline="2028-04-29 04:28:53.946157782 +0000 UTC"
Oct 30 19:30:52.489419 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:52.489416    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13112h58m1.456745502s"
Oct 30 19:30:52.541986 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:52.541917    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:30:52.542127 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:52.542038    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:30:52.576383 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:52.576340    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bxxz2" event={"ID":"35333d53-b864-432d-af6d-b1baec362af6","Type":"ContainerStarted","Data":"fe2bb906042c92e3f81991602e039cca05ede3fc631204df00ceba6e679577b5"}
Oct 30 19:30:52.579944 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:52.579909    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp" event={"ID":"6035c60e-c58e-455e-a61f-19eb599272af","Type":"ContainerStarted","Data":"f59a3d2fa1f4bdde3a8fe04e2021f2bfc6b47b7e33ecd49e248c1665a784d27b"}
Oct 30 19:30:52.584094 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:52.583344    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal" event={"ID":"2bdf808689747c42465325897cee67ca","Type":"ContainerStarted","Data":"b75f9978b8fef6a0601c6fd405797cd1b2260fec1308e353dd04acc0c6060162"}
Oct 30 19:30:52.590788 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:52.590714    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64whk" event={"ID":"ee3c157f-683e-415d-b856-74da8b4fe669","Type":"ContainerStarted","Data":"0e40c5103d22a472d26a9109f4f4897bb9cbb027fa6bbd4cc492602e19d223b2"}
Oct 30 19:30:52.594432 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:52.594362    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-45rm5" event={"ID":"d432cc60-02df-4a8d-94d4-93aafac213f2","Type":"ContainerStarted","Data":"deae8580d1249abde68c8681ce29ed814663d86a100c519e5bc3ca68198b88ed"}
Oct 30 19:30:52.599556 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:52.599334    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qnmn2" event={"ID":"0fef01e5-9be4-4368-8f1e-3f73d7d6c60d","Type":"ContainerStarted","Data":"3b5fcf354f1de5ae721c7e59bdbc57b009d0fcd8dffc411c85af9784f33c2318"}
Oct 30 19:30:52.600881 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:52.600845    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal" podStartSLOduration=2.600833565 podStartE2EDuration="2.600833565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:52.600764425 +0000 UTC m=+3.641631199" watchObservedRunningTime="2026-10-30 19:30:52.600833565 +0000 UTC m=+3.641700322"
Oct 30 19:30:52.604408 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:52.604249    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vjc5d" event={"ID":"cdfe62e2-9594-4c48-a3bc-2dce9df78b56","Type":"ContainerStarted","Data":"55ec1086b671b271d3dfa310dedfc2c6b40729d72cba4ca9141a1a58756155cd"}
Oct 30 19:30:52.608116 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:52.607974    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dwfmh" event={"ID":"ef93f664-f7a4-42b9-bb9c-a10c519e9685","Type":"ContainerStarted","Data":"5a1a321b62a9b9de7d0b59b69e9430c2a74a4f6b70b2d28b081282567e2300b4"}
Oct 30 19:30:52.613197 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:52.613074    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g" event={"ID":"5aad5010-0827-4918-85d9-47e337a22774","Type":"ContainerStarted","Data":"3ea2b374f029306389eb06c1bedfa13941608fc7e0fe310460a8d3b094709229"}
Oct 30 19:30:52.615690 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:52.615639    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4pwl6" event={"ID":"b16dc00a-c3b0-4767-a468-6d204ba44e55","Type":"ContainerStarted","Data":"85506832c693909051574640612a0a3de3e99ac27535720e9b9e5d5fcbd87580"}
Oct 30 19:30:53.019602 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:53.019288    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs\") pod \"network-metrics-daemon-xmzrn\" (UID: \"a8eb8c11-ccd0-457f-92bd-c21db1c89e60\") " pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:30:53.019765 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:53.019518    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:30:53.019765 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:53.019685    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs podName:a8eb8c11-ccd0-457f-92bd-c21db1c89e60 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:55.019665049 +0000 UTC m=+6.060531803 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs") pod "network-metrics-daemon-xmzrn" (UID: "a8eb8c11-ccd0-457f-92bd-c21db1c89e60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:30:53.120249 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:53.120210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:30:53.120446 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:53.120364    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:30:53.120446 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:53.120429    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret podName:b7a73dcc-6a78-4344-b0b3-7a8611466a41 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:55.120410844 +0000 UTC m=+6.161277611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret") pod "global-pull-secret-syncer-6vm8t" (UID: "b7a73dcc-6a78-4344-b0b3-7a8611466a41") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:30:53.221344 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:53.221307    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xn78\" (UniqueName: \"kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78\") pod \"network-check-target-hr4sv\" (UID: \"be888c82-964b-42ae-948a-e6e27d276347\") " pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:30:53.221525 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:53.221509    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:30:53.221592 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:53.221533    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:30:53.221592 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:53.221546    2576 projected.go:196] Error preparing data for projected volume kube-api-access-5xn78 for pod openshift-network-diagnostics/network-check-target-hr4sv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:30:53.221695 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:53.221602    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78 podName:be888c82-964b-42ae-948a-e6e27d276347 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:55.221585161 +0000 UTC m=+6.262451916 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5xn78" (UniqueName: "kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78") pod "network-check-target-hr4sv" (UID: "be888c82-964b-42ae-948a-e6e27d276347") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:30:53.545585 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:53.545554    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:30:53.546122 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:53.545686    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:30:53.546122 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:53.546030    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:30:53.546122 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:53.546119    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:30:53.630834 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:53.630783    2576 generic.go:356] "Generic (PLEG): container finished" podID="b1c62c9f273bf0b4293082033ce5e525" containerID="ac0d3a23121a795127e9043d29dd14f983ba4cf8c27d503d082b18a2930d0014" exitCode=0
Oct 30 19:30:53.631482 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:53.631391    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal" event={"ID":"b1c62c9f273bf0b4293082033ce5e525","Type":"ContainerDied","Data":"ac0d3a23121a795127e9043d29dd14f983ba4cf8c27d503d082b18a2930d0014"}
Oct 30 19:30:54.541921 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:54.541892    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:30:54.542205 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:54.542009    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:30:54.646559 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:54.646527    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal" event={"ID":"b1c62c9f273bf0b4293082033ce5e525","Type":"ContainerStarted","Data":"487d2240e2fa53ecdc8599bbb1e0d686e0400c70a9c5c78c041056893b0af133"}
Oct 30 19:30:54.671516 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:54.670726    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal" podStartSLOduration=4.670707275 podStartE2EDuration="4.670707275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:54.666803855 +0000 UTC m=+5.707670620" watchObservedRunningTime="2026-10-30 19:30:54.670707275 +0000 UTC m=+5.711574039"
Oct 30 19:30:55.041064 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:55.040966    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs\") pod \"network-metrics-daemon-xmzrn\" (UID: \"a8eb8c11-ccd0-457f-92bd-c21db1c89e60\") " pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:30:55.041199 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:55.041171    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:30:55.041246 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:55.041225    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs podName:a8eb8c11-ccd0-457f-92bd-c21db1c89e60 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:59.041211675 +0000 UTC m=+10.082078416 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs") pod "network-metrics-daemon-xmzrn" (UID: "a8eb8c11-ccd0-457f-92bd-c21db1c89e60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:30:55.141602 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:55.141563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:30:55.141788 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:55.141762    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:30:55.141850 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:55.141837    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret podName:b7a73dcc-6a78-4344-b0b3-7a8611466a41 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:59.141808186 +0000 UTC m=+10.182674933 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret") pod "global-pull-secret-syncer-6vm8t" (UID: "b7a73dcc-6a78-4344-b0b3-7a8611466a41") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:30:55.241975 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:55.241938    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xn78\" (UniqueName: \"kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78\") pod \"network-check-target-hr4sv\" (UID: \"be888c82-964b-42ae-948a-e6e27d276347\") " pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:30:55.242235 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:55.242149    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:30:55.242235 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:55.242171    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:30:55.242235 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:55.242187    2576 projected.go:196] Error preparing data for projected volume kube-api-access-5xn78 for pod openshift-network-diagnostics/network-check-target-hr4sv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:30:55.242527 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:55.242245    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78 podName:be888c82-964b-42ae-948a-e6e27d276347 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:59.242227072 +0000 UTC m=+10.283093827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5xn78" (UniqueName: "kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78") pod "network-check-target-hr4sv" (UID: "be888c82-964b-42ae-948a-e6e27d276347") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:30:55.545076 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:55.544993    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:30:55.545262 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:55.545156    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:30:55.545700 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:55.545663    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:30:55.545820 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:55.545763    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:30:55.663596 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:55.663558    2576 generic.go:356] "Generic (PLEG): container finished" podID="ee3c157f-683e-415d-b856-74da8b4fe669" containerID="4cede867c67d294b4cb8f31476c8b3bd5c1827a5b15c9dce8e8bdb17fd0b47a6" exitCode=0
Oct 30 19:30:55.664735 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:55.664708    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64whk" event={"ID":"ee3c157f-683e-415d-b856-74da8b4fe669","Type":"ContainerDied","Data":"4cede867c67d294b4cb8f31476c8b3bd5c1827a5b15c9dce8e8bdb17fd0b47a6"}
Oct 30 19:30:56.542322 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:56.542279    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:30:56.542499 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:56.542455    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:30:56.669126 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:56.669060    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qnmn2" event={"ID":"0fef01e5-9be4-4368-8f1e-3f73d7d6c60d","Type":"ContainerStarted","Data":"8e40911af80e8392fd391b2a7dbcabfe992d1c0ff6c5f668aa6a6af4566352af"}
Oct 30 19:30:56.687936 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:56.687874    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qnmn2" podStartSLOduration=3.859792296 podStartE2EDuration="7.687857407s" totalImagesPullingTime="3.828065111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:56.686881557 +0000 UTC m=+7.727748323" watchObservedRunningTime="2026-10-30 19:30:56.687857407 +0000 UTC m=+7.728724172"
Oct 30 19:30:57.543967 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:57.543944    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:30:57.544081 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:57.544057    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:30:57.544125 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:57.544103    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:30:57.544169 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:57.544154    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:30:57.673768 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:57.673685    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bxxz2" event={"ID":"35333d53-b864-432d-af6d-b1baec362af6","Type":"ContainerStarted","Data":"1534fbd3cdc387c6b51ecf8d4f5a1b832dd2fd6fee9bc798bb15132149cc62ec"}
Oct 30 19:30:57.678729 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:57.678699    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp" event={"ID":"6035c60e-c58e-455e-a61f-19eb599272af","Type":"ContainerStarted","Data":"ccc8b8495a7e704dca4cd8caf3783525a8246381e939f9a4881d9f61d3fa09df"}
Oct 30 19:30:58.157099 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:58.157063    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qnmn2"
Oct 30 19:30:58.157605 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:58.157584    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qnmn2"
Oct 30 19:30:58.175773 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:58.175717    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bxxz2" podStartSLOduration=3.926617546 podStartE2EDuration="9.175699691s" totalImagesPullingTime="5.249082145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:57.691922186 +0000 UTC m=+8.732788950" watchObservedRunningTime="2026-10-30 19:30:58.175699691 +0000 UTC m=+9.216566457"
Oct 30 19:30:58.542603 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:58.542564    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:30:58.542771 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:58.542727    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:30:59.076110 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:59.076076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs\") pod \"network-metrics-daemon-xmzrn\" (UID: \"a8eb8c11-ccd0-457f-92bd-c21db1c89e60\") " pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:30:59.076422 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:59.076279    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:30:59.076422 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:59.076342    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs podName:a8eb8c11-ccd0-457f-92bd-c21db1c89e60 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:07.07632504 +0000 UTC m=+18.117191794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs") pod "network-metrics-daemon-xmzrn" (UID: "a8eb8c11-ccd0-457f-92bd-c21db1c89e60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:30:59.080119 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:59.079934    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:30:59.177501 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:59.177428    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:30:59.177656 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:59.177580    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:30:59.177656 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:59.177649    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret podName:b7a73dcc-6a78-4344-b0b3-7a8611466a41 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:07.177631982 +0000 UTC m=+18.218498729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret") pod "global-pull-secret-syncer-6vm8t" (UID: "b7a73dcc-6a78-4344-b0b3-7a8611466a41") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:30:59.278712 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:59.278347    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xn78\" (UniqueName: \"kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78\") pod \"network-check-target-hr4sv\" (UID: \"be888c82-964b-42ae-948a-e6e27d276347\") " pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:30:59.278712 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:59.278495    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:30:59.278712 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:59.278519    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:30:59.278712 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:59.278531    2576 projected.go:196] Error preparing data for projected volume kube-api-access-5xn78 for pod openshift-network-diagnostics/network-check-target-hr4sv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:30:59.278712 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:59.278593    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78 podName:be888c82-964b-42ae-948a-e6e27d276347 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:07.278573386 +0000 UTC m=+18.319440142 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5xn78" (UniqueName: "kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78") pod "network-check-target-hr4sv" (UID: "be888c82-964b-42ae-948a-e6e27d276347") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:30:59.456568 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:59.456409    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:30:59.079952077Z","UUID":"32ac8b2d-ad9a-41e9-8cbe-613ae95d66d8","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:30:59.458033 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:59.458006    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:30:59.458033 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:59.458038    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:30:59.545672 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:59.545193    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:30:59.545672 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:59.545323    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:30:59.545996 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:59.545973    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:30:59.546126 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:30:59.546107    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:30:59.689166 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:59.689136    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp" event={"ID":"6035c60e-c58e-455e-a61f-19eb599272af","Type":"ContainerStarted","Data":"670925aa80c7c9582a9e05bb427fe3c1d63ff7423a5ced274fa50436c5ab72cb"}
Oct 30 19:30:59.691911 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:59.691334    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:30:59.691911 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:59.691378    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vjc5d" event={"ID":"cdfe62e2-9594-4c48-a3bc-2dce9df78b56","Type":"ContainerStarted","Data":"a24ce6c982b058a5d026d508ef5a5897e07626099f7d9f4f0d5de9a19a6b9da2"}
Oct 30 19:30:59.716175 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:30:59.713377    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-vjc5d" podStartSLOduration=3.212340544 podStartE2EDuration="10.713258776s" totalImagesPullingTime="7.500918232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:59.712209263 +0000 UTC m=+10.753076026" watchObservedRunningTime="2026-10-30 19:30:59.713258776 +0000 UTC m=+10.754125540"
Oct 30 19:31:00.542178 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:00.542144    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:00.542566 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:00.542262    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:00.695548 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:00.695082    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dwfmh" event={"ID":"ef93f664-f7a4-42b9-bb9c-a10c519e9685","Type":"ContainerStarted","Data":"cae5fa2285fe00a4af807c6075ef350abf861a1faf299280f69530d3b553720c"}
Oct 30 19:31:00.697610 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:00.697560    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp" event={"ID":"6035c60e-c58e-455e-a61f-19eb599272af","Type":"ContainerStarted","Data":"b7654c2ef691458f205a826d85698c57d0ada89b24b88fcdd8246455f1abe726"}
Oct 30 19:31:00.701686 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:00.701658    2576 generic.go:356] "Generic (PLEG): container finished" podID="ee3c157f-683e-415d-b856-74da8b4fe669" containerID="291ffdce5b0e571e858c4783ae9e8295b6fd0919af05e254934a075cb06e962b" exitCode=0
Oct 30 19:31:00.701801 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:00.701753    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64whk" event={"ID":"ee3c157f-683e-415d-b856-74da8b4fe669","Type":"ContainerDied","Data":"291ffdce5b0e571e858c4783ae9e8295b6fd0919af05e254934a075cb06e962b"}
Oct 30 19:31:00.716419 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:00.716363    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-dwfmh" podStartSLOduration=3.763689651 podStartE2EDuration="11.716345407s" totalImagesPullingTime="7.952655756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:00.716237328 +0000 UTC m=+11.757104115" watchObservedRunningTime="2026-10-30 19:31:00.716345407 +0000 UTC m=+11.757212171"
Oct 30 19:31:00.738300 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:00.738258    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp" podStartSLOduration=3.600825798 podStartE2EDuration="11.738244193s" totalImagesPullingTime="8.137418395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:00.737639318 +0000 UTC m=+11.778506082" watchObservedRunningTime="2026-10-30 19:31:00.738244193 +0000 UTC m=+11.779110955"
Oct 30 19:31:01.541929 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:01.541698    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:01.542101 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:01.542070    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:01.542547 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:01.542524    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:01.542985 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:01.542632    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:01.706609 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:01.706522    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64whk" event={"ID":"ee3c157f-683e-415d-b856-74da8b4fe669","Type":"ContainerStarted","Data":"02d24d5b5a11a7daff8c46ae0c2e5145de8b599afb53a33e821abb1be5fd7fbc"}
Oct 30 19:31:01.708425 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:01.708398    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-45rm5" event={"ID":"d432cc60-02df-4a8d-94d4-93aafac213f2","Type":"ContainerStarted","Data":"b4142e6d212ded578d62dc336181dfc8ea2dca50865a3fe2d0cf7c2c606eca99"}
Oct 30 19:31:01.746948 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:01.746901    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-45rm5" podStartSLOduration=4.746231311 podStartE2EDuration="12.746883487s" totalImagesPullingTime="8.000652176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:01.746669436 +0000 UTC m=+12.787536199" watchObservedRunningTime="2026-10-30 19:31:01.746883487 +0000 UTC m=+12.787750250"
Oct 30 19:31:02.541600 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:02.541566    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:02.541789 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:02.541757    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:02.712543 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:02.712507    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4pwl6" event={"ID":"b16dc00a-c3b0-4767-a468-6d204ba44e55","Type":"ContainerStarted","Data":"ec7e62b47ec10e5f2745afc6f1f977d21ef1b70992e2adacfa5533d0c2bdf2ed"}
Oct 30 19:31:02.715174 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:02.715147    2576 generic.go:356] "Generic (PLEG): container finished" podID="ee3c157f-683e-415d-b856-74da8b4fe669" containerID="02d24d5b5a11a7daff8c46ae0c2e5145de8b599afb53a33e821abb1be5fd7fbc" exitCode=0
Oct 30 19:31:02.715981 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:02.715947    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64whk" event={"ID":"ee3c157f-683e-415d-b856-74da8b4fe669","Type":"ContainerDied","Data":"02d24d5b5a11a7daff8c46ae0c2e5145de8b599afb53a33e821abb1be5fd7fbc"}
Oct 30 19:31:02.732788 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:02.732732    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4pwl6" podStartSLOduration=3.673674257 podStartE2EDuration="13.732714565s" totalImagesPullingTime="10.059040308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:02.732511992 +0000 UTC m=+13.773378755" watchObservedRunningTime="2026-10-30 19:31:02.732714565 +0000 UTC m=+13.773581333"
Oct 30 19:31:03.541785 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:03.541755    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:03.542403 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:03.542365    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:03.542473 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:03.542411    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:03.542534 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:03.542516    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:03.720598 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:03.720556    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64whk" event={"ID":"ee3c157f-683e-415d-b856-74da8b4fe669","Type":"ContainerStarted","Data":"d2317b6b9331f8979be1a78659d9111859a8e1ce85fdbdd59c0276d8404924e8"}
Oct 30 19:31:04.541713 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:04.541678    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:04.541880 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:04.541800    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:04.725891 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:04.725851    2576 generic.go:356] "Generic (PLEG): container finished" podID="ee3c157f-683e-415d-b856-74da8b4fe669" containerID="d2317b6b9331f8979be1a78659d9111859a8e1ce85fdbdd59c0276d8404924e8" exitCode=0
Oct 30 19:31:04.726544 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:04.725915    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64whk" event={"ID":"ee3c157f-683e-415d-b856-74da8b4fe669","Type":"ContainerDied","Data":"d2317b6b9331f8979be1a78659d9111859a8e1ce85fdbdd59c0276d8404924e8"}
Oct 30 19:31:05.542686 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:05.542120    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:05.542686 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:05.542153    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:05.542686 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:05.542275    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:05.542686 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:05.542642    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:06.541974 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:06.541816    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:06.542327 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:06.542113    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:06.732901 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:06.732877    2576 generic.go:356] "Generic (PLEG): container finished" podID="ee3c157f-683e-415d-b856-74da8b4fe669" containerID="49cbc617eec9bfa372e6a9840a85c7f1ee504f2971f1d6dbbeeb406f1e40c1c4" exitCode=0
Oct 30 19:31:06.732984 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:06.732947    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64whk" event={"ID":"ee3c157f-683e-415d-b856-74da8b4fe669","Type":"ContainerDied","Data":"49cbc617eec9bfa372e6a9840a85c7f1ee504f2971f1d6dbbeeb406f1e40c1c4"}
Oct 30 19:31:06.735584 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:06.735556    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g" event={"ID":"5aad5010-0827-4918-85d9-47e337a22774","Type":"ContainerStarted","Data":"d564a6ff9e6febeb4ec12f1bce59ec63525a1c3b5031b54de856b426d1b7f354"}
Oct 30 19:31:06.735681 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:06.735657    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g" event={"ID":"5aad5010-0827-4918-85d9-47e337a22774","Type":"ContainerStarted","Data":"3200c3d6e5fb4115555d729529a2efa537ac1d26dc941cb2f8e2a473d7d02921"}
Oct 30 19:31:06.735768 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:06.735738    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g" event={"ID":"5aad5010-0827-4918-85d9-47e337a22774","Type":"ContainerStarted","Data":"d3d7b299ddffd1994bcd799735c19b8242c5b8417242bf478e155bcc740aac7d"}
Oct 30 19:31:07.135014 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:07.134797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs\") pod \"network-metrics-daemon-xmzrn\" (UID: \"a8eb8c11-ccd0-457f-92bd-c21db1c89e60\") " pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:07.135014 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:07.134935    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:31:07.135203 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:07.135024    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs podName:a8eb8c11-ccd0-457f-92bd-c21db1c89e60 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:23.135010469 +0000 UTC m=+34.175877224 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs") pod "network-metrics-daemon-xmzrn" (UID: "a8eb8c11-ccd0-457f-92bd-c21db1c89e60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:31:07.235978 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:07.235942    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:07.236163 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:07.236081    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:31:07.236163 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:07.236131    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret podName:b7a73dcc-6a78-4344-b0b3-7a8611466a41 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:23.236119723 +0000 UTC m=+34.276986478 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret") pod "global-pull-secret-syncer-6vm8t" (UID: "b7a73dcc-6a78-4344-b0b3-7a8611466a41") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:31:07.337122 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:07.337078    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xn78\" (UniqueName: \"kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78\") pod \"network-check-target-hr4sv\" (UID: \"be888c82-964b-42ae-948a-e6e27d276347\") " pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:07.337315 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:07.337265    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:31:07.337315 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:07.337288    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:31:07.337315 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:07.337299    2576 projected.go:196] Error preparing data for projected volume kube-api-access-5xn78 for pod openshift-network-diagnostics/network-check-target-hr4sv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:31:07.337458 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:07.337362    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78 podName:be888c82-964b-42ae-948a-e6e27d276347 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:23.337344668 +0000 UTC m=+34.378211426 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5xn78" (UniqueName: "kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78") pod "network-check-target-hr4sv" (UID: "be888c82-964b-42ae-948a-e6e27d276347") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:31:07.542664 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:07.542574    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:07.542664 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:07.542605    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:07.543212 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:07.542695    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:07.543212 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:07.542790    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:07.740690 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:07.740653    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g" event={"ID":"5aad5010-0827-4918-85d9-47e337a22774","Type":"ContainerStarted","Data":"b79948969e61d0c7d0ceda9009b8ab6d28f4721f0bbe5af46249c15998e3eb5f"}
Oct 30 19:31:07.740884 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:07.740726    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g" event={"ID":"5aad5010-0827-4918-85d9-47e337a22774","Type":"ContainerStarted","Data":"b8510a82a9d3e3d4d998b800ddf0224e1bf97737254adc5d902d377ca76cf350"}
Oct 30 19:31:07.740884 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:07.740762    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g" event={"ID":"5aad5010-0827-4918-85d9-47e337a22774","Type":"ContainerStarted","Data":"5aadfea3cb0d6807f6f0d376827155e95d78974fe5c623d0fe63408d63482572"}
Oct 30 19:31:07.743396 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:07.743372    2576 generic.go:356] "Generic (PLEG): container finished" podID="ee3c157f-683e-415d-b856-74da8b4fe669" containerID="6a9bd36648ae2133d8b9d1cf320819f238f297e0950ddd16c5ed756c79a5e94d" exitCode=0
Oct 30 19:31:07.743491 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:07.743403    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64whk" event={"ID":"ee3c157f-683e-415d-b856-74da8b4fe669","Type":"ContainerDied","Data":"6a9bd36648ae2133d8b9d1cf320819f238f297e0950ddd16c5ed756c79a5e94d"}
Oct 30 19:31:08.542416 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:08.542379    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:08.542616 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:08.542504    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:08.748367 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:08.748325    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64whk" event={"ID":"ee3c157f-683e-415d-b856-74da8b4fe669","Type":"ContainerStarted","Data":"d432c5646ac8ec25f511a7d23c3a1d99d049c959fe1100a037594566aa648658"}
Oct 30 19:31:08.779065 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:08.777735    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-64whk" podStartSLOduration=9.150327788 podStartE2EDuration="19.777717376s" totalImagesPullingTime="10.627389588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:08.776890546 +0000 UTC m=+19.817757322" watchObservedRunningTime="2026-10-30 19:31:08.777717376 +0000 UTC m=+19.818584140"
Oct 30 19:31:09.543062 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:09.543026    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:09.543226 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:09.543149    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:09.543226 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:09.543201    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:09.543317 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:09.543299    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:09.753674 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:09.753642    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g" event={"ID":"5aad5010-0827-4918-85d9-47e337a22774","Type":"ContainerStarted","Data":"42123150e16d8e20ea5359e37d99e11ec15ae97561f4e4442eba2a005a3c5a83"}
Oct 30 19:31:10.542185 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:10.542153    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:10.542321 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:10.542244    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:11.542176 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:11.542141    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:11.542176 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:11.542160    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:11.543186 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:11.542245    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:11.543186 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:11.542320    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:11.760056 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:11.760006    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g" event={"ID":"5aad5010-0827-4918-85d9-47e337a22774","Type":"ContainerStarted","Data":"1e4a1198a1cf10691ec2019334c10ff0885bd9bc3afe07c476ab47adbde72845"}
Oct 30 19:31:11.760331 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:11.760313    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:31:11.760436 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:11.760391    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:31:11.776222 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:11.776191    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:31:11.796365 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:11.796278    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g" podStartSLOduration=8.026220854 podStartE2EDuration="22.79626502s" totalImagesPullingTime="14.770044166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:11.79571675 +0000 UTC m=+22.836583516" watchObservedRunningTime="2026-10-30 19:31:11.79626502 +0000 UTC m=+22.837131801"
Oct 30 19:31:12.541961 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:12.541710    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:12.542233 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:12.542102    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:12.763461 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:12.763416    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:31:12.782066 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:12.782020    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:31:12.836394 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:12.836297    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6vm8t"]
Oct 30 19:31:12.836523 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:12.836406    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:12.836523 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:12.836495    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:12.839546 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:12.839516    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xmzrn"]
Oct 30 19:31:12.839689 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:12.839628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:12.839772 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:12.839744    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:12.840254 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:12.840235    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hr4sv"]
Oct 30 19:31:12.840357 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:12.840311    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:12.840394 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:12.840384    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:14.541831 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:14.541800    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:14.542301 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:14.541867    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:14.542301 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:14.541959    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:14.542301 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:14.541975    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:14.542301 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:14.542069    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:14.542301 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:14.542099    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:15.108107 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:15.108080    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qnmn2"
Oct 30 19:31:15.108287 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:15.108193    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:31:15.108871 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:15.108839    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qnmn2"
Oct 30 19:31:16.542404 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:16.542373    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:16.542766 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:16.542373    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:16.542766 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:16.542487    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:16.542766 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:16.542373    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:16.542766 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:16.542549    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:16.542766 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:16.542643    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:18.542246 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:18.542207    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:18.542246 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:18.542232    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:18.542246 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:18.542243    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:18.542702 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:18.542338    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:18.542702 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:18.542447    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:18.542702 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:18.542508    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:19.156981 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:19.156959    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dwfmh_ef93f664-f7a4-42b9-bb9c-a10c519e9685/dns-node-resolver/0.log"
Oct 30 19:31:19.724959 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:19.724930    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bxxz2_35333d53-b864-432d-af6d-b1baec362af6/node-ca/0.log"
Oct 30 19:31:20.542139 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:20.542106    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:20.542139 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:20.542137    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:20.542360 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:20.542100    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:20.542360 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:20.542233    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:20.542360 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:20.542288    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:20.542360 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:20.542341    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:22.542467 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:22.542439    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:22.542941 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:22.542538    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:22.542941 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:22.542546    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:22.542941 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:22.542680    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:22.542941 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:22.542733    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:22.542941 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:22.542810    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:23.142439 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:23.142418    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs\") pod \"network-metrics-daemon-xmzrn\" (UID: \"a8eb8c11-ccd0-457f-92bd-c21db1c89e60\") " pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:23.142590 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:23.142572    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:31:23.142643 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:23.142634    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs podName:a8eb8c11-ccd0-457f-92bd-c21db1c89e60 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:55.142619006 +0000 UTC m=+66.183485751 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs") pod "network-metrics-daemon-xmzrn" (UID: "a8eb8c11-ccd0-457f-92bd-c21db1c89e60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:31:23.243252 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:23.243226    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:23.243396 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:23.243331    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:31:23.243396 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:23.243377    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret podName:b7a73dcc-6a78-4344-b0b3-7a8611466a41 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:55.243364562 +0000 UTC m=+66.284231304 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret") pod "global-pull-secret-syncer-6vm8t" (UID: "b7a73dcc-6a78-4344-b0b3-7a8611466a41") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:31:23.344033 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:23.344005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xn78\" (UniqueName: \"kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78\") pod \"network-check-target-hr4sv\" (UID: \"be888c82-964b-42ae-948a-e6e27d276347\") " pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:23.344196 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:23.344137    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:31:23.344196 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:23.344150    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:31:23.344196 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:23.344158    2576 projected.go:196] Error preparing data for projected volume kube-api-access-5xn78 for pod openshift-network-diagnostics/network-check-target-hr4sv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:31:23.344292 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:23.344199    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78 podName:be888c82-964b-42ae-948a-e6e27d276347 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:55.344188225 +0000 UTC m=+66.385054980 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5xn78" (UniqueName: "kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78") pod "network-check-target-hr4sv" (UID: "be888c82-964b-42ae-948a-e6e27d276347") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:31:24.541860 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:24.541821    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:24.541860 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:24.541847    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:24.542313 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:24.541973    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:24.542313 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:24.542016    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:24.542313 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:24.542122    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:24.542313 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:24.542194    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:26.542208 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:26.542174    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:26.542691 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:26.542174    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:26.542691 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:26.542303    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:26.542691 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:26.542174    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:26.542691 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:26.542356    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:26.542691 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:26.542425    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:28.542335 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:28.542299    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:28.542335 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:28.542328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:28.542808 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:28.542399    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:28.542808 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:28.542438    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:28.542808 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:28.542504    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:28.542808 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:28.542616    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:30.541727 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:30.541688    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:30.542171 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:30.541756    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:30.542171 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:30.541852    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:30.542171 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:30.541858    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:30.542171 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:30.541911    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:30.542171 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:30.541978    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:32.541788 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:32.541743    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:32.541788 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:32.541777    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:32.542349 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:32.541882    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:32.542349 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:32.541950    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:32.542349 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:32.541997    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:32.542349 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:32.542091    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:34.541585 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:34.541548    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:34.541585 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:34.541581    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:34.542016 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:34.541653    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:34.542016 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:34.541673    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vm8t" podUID="b7a73dcc-6a78-4344-b0b3-7a8611466a41"
Oct 30 19:31:34.542016 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:34.541721    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hr4sv" podUID="be888c82-964b-42ae-948a-e6e27d276347"
Oct 30 19:31:34.542016 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:34.541814    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmzrn" podUID="a8eb8c11-ccd0-457f-92bd-c21db1c89e60"
Oct 30 19:31:35.727872 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.727845    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeReady"
Oct 30 19:31:35.728247 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.727959    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 19:31:35.783682 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.783647    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pzxdj"]
Oct 30 19:31:35.786582 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.785883    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xbxhg"]
Oct 30 19:31:35.786773 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.786752    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:31:35.788523 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.788499    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:31:35.791662 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.791629    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:31:35.791762 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.791695    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:31:35.791995 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.791979    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h46gl\""
Oct 30 19:31:35.791995 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.791979    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:31:35.795074 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.795037    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:31:35.796654 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.796622    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pzxdj"]
Oct 30 19:31:35.799742 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.799720    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xbxhg"]
Oct 30 19:31:35.808993 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.808969    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-crfs6"]
Oct 30 19:31:35.810554 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.810540    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-crfs6"
Oct 30 19:31:35.823257 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.823237    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-crfs6"]
Oct 30 19:31:35.892256 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.892180    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:31:35.892521 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.892490    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-scxl6\""
Oct 30 19:31:35.892521 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.892518    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f2mgm\""
Oct 30 19:31:35.892673 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.892522    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:31:35.892673 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.892561    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:31:35.892673 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.892562    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:31:35.892673 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.892593    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:31:35.927410 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.927381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cedda70a-9baa-4592-818a-63092c854c8e-data-volume\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:31:35.927539 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.927423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqggx\" (UniqueName: \"kubernetes.io/projected/a675f8c3-6d56-485d-8c87-438d66ebdc1e-kube-api-access-xqggx\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:31:35.927539 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.927509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a675f8c3-6d56-485d-8c87-438d66ebdc1e-config-volume\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:31:35.927539 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.927530    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cedda70a-9baa-4592-818a-63092c854c8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:31:35.927678 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.927556    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfda46fc-34fd-461d-a2bf-d4ec19596ed2-cert\") pod \"ingress-canary-crfs6\" (UID: \"cfda46fc-34fd-461d-a2bf-d4ec19596ed2\") " pod="openshift-ingress-canary/ingress-canary-crfs6"
Oct 30 19:31:35.927678 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.927573    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vpzr\" (UniqueName: \"kubernetes.io/projected/cfda46fc-34fd-461d-a2bf-d4ec19596ed2-kube-api-access-8vpzr\") pod \"ingress-canary-crfs6\" (UID: \"cfda46fc-34fd-461d-a2bf-d4ec19596ed2\") " pod="openshift-ingress-canary/ingress-canary-crfs6"
Oct 30 19:31:35.927678 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.927614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cedda70a-9baa-4592-818a-63092c854c8e-crio-socket\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:31:35.927763 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.927683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a675f8c3-6d56-485d-8c87-438d66ebdc1e-tmp-dir\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:31:35.927763 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.927735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t74h\" (UniqueName: \"kubernetes.io/projected/cedda70a-9baa-4592-818a-63092c854c8e-kube-api-access-4t74h\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:31:35.927763 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.927760    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a675f8c3-6d56-485d-8c87-438d66ebdc1e-metrics-tls\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:31:35.927849 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:35.927816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cedda70a-9baa-4592-818a-63092c854c8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:31:36.028737 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.028704    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t74h\" (UniqueName: \"kubernetes.io/projected/cedda70a-9baa-4592-818a-63092c854c8e-kube-api-access-4t74h\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:31:36.028737 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.028742    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a675f8c3-6d56-485d-8c87-438d66ebdc1e-metrics-tls\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:31:36.028905 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.028766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cedda70a-9baa-4592-818a-63092c854c8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:31:36.028905 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.028797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cedda70a-9baa-4592-818a-63092c854c8e-data-volume\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:31:36.028905 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.028835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqggx\" (UniqueName: \"kubernetes.io/projected/a675f8c3-6d56-485d-8c87-438d66ebdc1e-kube-api-access-xqggx\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:31:36.028905 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.028874    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a675f8c3-6d56-485d-8c87-438d66ebdc1e-config-volume\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:31:36.028905 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.028900    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cedda70a-9baa-4592-818a-63092c854c8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:31:36.029180 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.028934    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfda46fc-34fd-461d-a2bf-d4ec19596ed2-cert\") pod \"ingress-canary-crfs6\" (UID: \"cfda46fc-34fd-461d-a2bf-d4ec19596ed2\") " pod="openshift-ingress-canary/ingress-canary-crfs6"
Oct 30 19:31:36.029180 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.028957    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vpzr\" (UniqueName: \"kubernetes.io/projected/cfda46fc-34fd-461d-a2bf-d4ec19596ed2-kube-api-access-8vpzr\") pod \"ingress-canary-crfs6\" (UID: \"cfda46fc-34fd-461d-a2bf-d4ec19596ed2\") " pod="openshift-ingress-canary/ingress-canary-crfs6"
Oct 30 19:31:36.029180 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.028984    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cedda70a-9baa-4592-818a-63092c854c8e-crio-socket\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:31:36.029180 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.029011    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a675f8c3-6d56-485d-8c87-438d66ebdc1e-tmp-dir\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:31:36.029396 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.029366    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cedda70a-9baa-4592-818a-63092c854c8e-crio-socket\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:31:36.030169 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.030144    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cedda70a-9baa-4592-818a-63092c854c8e-data-volume\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:31:36.030280 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.030170    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a675f8c3-6d56-485d-8c87-438d66ebdc1e-tmp-dir\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:31:36.030391 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.030368    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a675f8c3-6d56-485d-8c87-438d66ebdc1e-config-volume\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:31:36.030455 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.030434    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cedda70a-9baa-4592-818a-63092c854c8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:31:36.032930 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.032906    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a675f8c3-6d56-485d-8c87-438d66ebdc1e-metrics-tls\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:31:36.033005 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.032928    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cedda70a-9baa-4592-818a-63092c854c8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:31:36.033005 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.032987    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfda46fc-34fd-461d-a2bf-d4ec19596ed2-cert\") pod \"ingress-canary-crfs6\" (UID: \"cfda46fc-34fd-461d-a2bf-d4ec19596ed2\") " pod="openshift-ingress-canary/ingress-canary-crfs6"
Oct 30 19:31:36.042416 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.042391    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t74h\" (UniqueName: \"kubernetes.io/projected/cedda70a-9baa-4592-818a-63092c854c8e-kube-api-access-4t74h\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:31:36.042529 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.042513    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqggx\" (UniqueName: \"kubernetes.io/projected/a675f8c3-6d56-485d-8c87-438d66ebdc1e-kube-api-access-xqggx\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:31:36.042755 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.042733    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vpzr\" (UniqueName: \"kubernetes.io/projected/cfda46fc-34fd-461d-a2bf-d4ec19596ed2-kube-api-access-8vpzr\") pod \"ingress-canary-crfs6\" (UID: \"cfda46fc-34fd-461d-a2bf-d4ec19596ed2\") " pod="openshift-ingress-canary/ingress-canary-crfs6"
Oct 30 19:31:36.101622 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.101588    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:31:36.108930 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.108905    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:31:36.119615 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.119594    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-crfs6"
Oct 30 19:31:36.393436 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.393415    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xbxhg"]
Oct 30 19:31:36.395800 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.395778    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pzxdj"]
Oct 30 19:31:36.398137 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:31:36.398112    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda675f8c3_6d56_485d_8c87_438d66ebdc1e.slice/crio-4d2ce302416f14e67ff465ce723e3d0a21e9f1859dd8e1b26044efe09af53447 WatchSource:0}: Error finding container 4d2ce302416f14e67ff465ce723e3d0a21e9f1859dd8e1b26044efe09af53447: Status 404 returned error can't find the container with id 4d2ce302416f14e67ff465ce723e3d0a21e9f1859dd8e1b26044efe09af53447
Oct 30 19:31:36.401149 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.401127    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-crfs6"]
Oct 30 19:31:36.404491 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:31:36.404468    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfda46fc_34fd_461d_a2bf_d4ec19596ed2.slice/crio-99a447cd455cca759581ade9ee8a552300faff916254f55447b1a5ee5c6461a3 WatchSource:0}: Error finding container 99a447cd455cca759581ade9ee8a552300faff916254f55447b1a5ee5c6461a3: Status 404 returned error can't find the container with id 99a447cd455cca759581ade9ee8a552300faff916254f55447b1a5ee5c6461a3
Oct 30 19:31:36.541939 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.541906    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:36.542120 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.541937    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:36.542120 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.542090    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:36.546234 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.546214    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:31:36.546355 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.546302    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rpkw7\""
Oct 30 19:31:36.546405 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.546358    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:31:36.546750 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.546737    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jznq8\""
Oct 30 19:31:36.547082 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.547060    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:31:36.547172 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.547088    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wr97x\""
Oct 30 19:31:36.547172 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.547151    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:31:36.815624 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.815545    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-crfs6" event={"ID":"cfda46fc-34fd-461d-a2bf-d4ec19596ed2","Type":"ContainerStarted","Data":"99a447cd455cca759581ade9ee8a552300faff916254f55447b1a5ee5c6461a3"}
Oct 30 19:31:36.817003 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.816975    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xbxhg" event={"ID":"cedda70a-9baa-4592-818a-63092c854c8e","Type":"ContainerStarted","Data":"84cdc30429e07f187ad69c690d4b1fe8d42a5a52141643864b809d22d79699e3"}
Oct 30 19:31:36.817125 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.817038    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xbxhg" event={"ID":"cedda70a-9baa-4592-818a-63092c854c8e","Type":"ContainerStarted","Data":"70833a700389847c63f703c4b20bba947f3be67b4805ea677d0a1b085ec704dd"}
Oct 30 19:31:36.818602 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:36.818556    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pzxdj" event={"ID":"a675f8c3-6d56-485d-8c87-438d66ebdc1e","Type":"ContainerStarted","Data":"4d2ce302416f14e67ff465ce723e3d0a21e9f1859dd8e1b26044efe09af53447"}
Oct 30 19:31:37.541496 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:37.541412    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dwfmh_ef93f664-f7a4-42b9-bb9c-a10c519e9685/dns-node-resolver/0.log"
Oct 30 19:31:37.825532 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:37.825453    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xbxhg" event={"ID":"cedda70a-9baa-4592-818a-63092c854c8e","Type":"ContainerStarted","Data":"5c7a673b1e1dfdba32bb25b2dc8c15300fde1c0ad733f9b037cda45646221a55"}
Oct 30 19:31:38.021314 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.021282    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"]
Oct 30 19:31:38.023501 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.023475    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:31:38.034732 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.034364    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 19:31:38.034732 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.034505    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:31:38.034732 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.034611    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:31:38.034732 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.034692    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:31:38.035133 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.034769    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 19:31:38.035133 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.034926    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-q5q65\""
Oct 30 19:31:38.036096 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.035927    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"]
Oct 30 19:31:38.148244 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.148202    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6b01178-0262-47a5-81bb-d715f5cf4380-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:31:38.148432 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.148260    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6b01178-0262-47a5-81bb-d715f5cf4380-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:31:38.148432 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.148293    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvm7j\" (UniqueName: \"kubernetes.io/projected/f6b01178-0262-47a5-81bb-d715f5cf4380-kube-api-access-bvm7j\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:31:38.148432 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.148386    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6b01178-0262-47a5-81bb-d715f5cf4380-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:31:38.249399 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.249358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6b01178-0262-47a5-81bb-d715f5cf4380-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:31:38.249563 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.249433    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6b01178-0262-47a5-81bb-d715f5cf4380-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:31:38.249563 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.249463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6b01178-0262-47a5-81bb-d715f5cf4380-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:31:38.249563 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.249494    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvm7j\" (UniqueName: \"kubernetes.io/projected/f6b01178-0262-47a5-81bb-d715f5cf4380-kube-api-access-bvm7j\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:31:38.250844 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.250781    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6b01178-0262-47a5-81bb-d715f5cf4380-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:31:38.252890 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.252860    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6b01178-0262-47a5-81bb-d715f5cf4380-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:31:38.253101 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.253038    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6b01178-0262-47a5-81bb-d715f5cf4380-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:31:38.267674 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.267619    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvm7j\" (UniqueName: \"kubernetes.io/projected/f6b01178-0262-47a5-81bb-d715f5cf4380-kube-api-access-bvm7j\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:31:38.341339 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.341305    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:31:38.341997 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.341968    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bxxz2_35333d53-b864-432d-af6d-b1baec362af6/node-ca/0.log"
Oct 30 19:31:38.636524 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.636490    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"]
Oct 30 19:31:38.640889 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:31:38.640858    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b01178_0262_47a5_81bb_d715f5cf4380.slice/crio-09163586b4ffabebea3176275992c748d561840fdc4417388c063f201ca008ac WatchSource:0}: Error finding container 09163586b4ffabebea3176275992c748d561840fdc4417388c063f201ca008ac: Status 404 returned error can't find the container with id 09163586b4ffabebea3176275992c748d561840fdc4417388c063f201ca008ac
Oct 30 19:31:38.830605 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.830520    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5" event={"ID":"f6b01178-0262-47a5-81bb-d715f5cf4380","Type":"ContainerStarted","Data":"09163586b4ffabebea3176275992c748d561840fdc4417388c063f201ca008ac"}
Oct 30 19:31:38.832822 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.832789    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pzxdj" event={"ID":"a675f8c3-6d56-485d-8c87-438d66ebdc1e","Type":"ContainerStarted","Data":"d216ad641439bc15a3e7a0dae18e6b80a09c53dfcbb5fb379c6c0d55c18e6e07"}
Oct 30 19:31:38.832963 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.832843    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pzxdj" event={"ID":"a675f8c3-6d56-485d-8c87-438d66ebdc1e","Type":"ContainerStarted","Data":"27b6e280c6a68f0df701c4fc896c34782d3f016131336088ab19ccecd629671e"}
Oct 30 19:31:38.833139 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.833117    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:31:38.853846 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:38.853793    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-pzxdj" podStartSLOduration=1.965384091 podStartE2EDuration="3.853778669s" totalImagesPullingTime="1.888394578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:38.852388704 +0000 UTC m=+49.893255472" watchObservedRunningTime="2026-10-30 19:31:38.853778669 +0000 UTC m=+49.894645432"
Oct 30 19:31:39.837595 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:39.837550    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xbxhg" event={"ID":"cedda70a-9baa-4592-818a-63092c854c8e","Type":"ContainerStarted","Data":"843313ab4aaeaf33b3bfa9047952e24c1d232e9a2ea4d26092b008f9547c801b"}
Oct 30 19:31:39.839034 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:39.839006    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-crfs6" event={"ID":"cfda46fc-34fd-461d-a2bf-d4ec19596ed2","Type":"ContainerStarted","Data":"3cb2af7f43ad9c15a7d7070b829d96e5eabacd3cf97a23173f7692817cf899ed"}
Oct 30 19:31:39.859695 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:39.859652    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xbxhg" podStartSLOduration=2.467310316 podStartE2EDuration="4.859639325s" totalImagesPullingTime="2.392329009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:31:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:39.859175672 +0000 UTC m=+50.900042455" watchObservedRunningTime="2026-10-30 19:31:39.859639325 +0000 UTC m=+50.900506088"
Oct 30 19:31:39.886240 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:39.886193    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-crfs6" podStartSLOduration=2.022580373 podStartE2EDuration="4.886176359s" totalImagesPullingTime="2.863595986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:39.885763169 +0000 UTC m=+50.926629954" watchObservedRunningTime="2026-10-30 19:31:39.886176359 +0000 UTC m=+50.927043125"
Oct 30 19:31:40.844437 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:40.844394    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5" event={"ID":"f6b01178-0262-47a5-81bb-d715f5cf4380","Type":"ContainerStarted","Data":"54e976d2b3e992383eaa8a00a734f85c7221eebb0288f4b2baabf733183a64d8"}
Oct 30 19:31:40.844799 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:40.844482    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5" event={"ID":"f6b01178-0262-47a5-81bb-d715f5cf4380","Type":"ContainerStarted","Data":"2e0fa468fc3b2e8a6ce53f879ed4e060e1b687391508c069533e0be89a8bd714"}
Oct 30 19:31:40.866072 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:40.866011    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5" podStartSLOduration=2.664375283 podStartE2EDuration="3.865997828s" totalImagesPullingTime="1.201622545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:40.865479482 +0000 UTC m=+51.906346271" watchObservedRunningTime="2026-10-30 19:31:40.865997828 +0000 UTC m=+51.906864590"
Oct 30 19:31:42.433067 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.433023    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"]
Oct 30 19:31:42.436212 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.436193    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.442956 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.442933    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4jpzb"]
Oct 30 19:31:42.445857 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.445808    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.448346 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.448326    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"]
Oct 30 19:31:42.492822 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.492795    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:31:42.493504 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.493074    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 19:31:42.493504 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.493165    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:31:42.499751 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.499727    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6xbb8\""
Oct 30 19:31:42.499861 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.499813    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 19:31:42.499913 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.499857    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vzztc\""
Oct 30 19:31:42.518957 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.518930    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:31:42.519310 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.519287    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 19:31:42.596128 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.596081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.596400 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.596156    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-tls\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.596400 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.596228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.596400 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.596336    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4201a324-4bf9-46fc-b1be-c8381f3e9dba-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.596400 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.596366    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4jj8\" (UniqueName: \"kubernetes.io/projected/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-api-access-g4jj8\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.596586 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.596408    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-sys\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.596586 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.596461    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.596586 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.596542    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-wtmp\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.596586 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.596569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-accelerators-collector-config\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.596762 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.596606    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-root\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.596762 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.596669    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4201a324-4bf9-46fc-b1be-c8381f3e9dba-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.596762 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.596700    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl4qt\" (UniqueName: \"kubernetes.io/projected/92d09c10-51ac-4231-8195-d0c853887162-kube-api-access-cl4qt\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.596762 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.596726    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92d09c10-51ac-4231-8195-d0c853887162-metrics-client-ca\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.596762 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.596752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-textfile\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.596894 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.596826    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.697526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.697427    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-textfile\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.697526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.697480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.697526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.697521    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.697526 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.697538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-tls\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.697830 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.697553    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.697830 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.697569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4201a324-4bf9-46fc-b1be-c8381f3e9dba-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.697830 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.697590    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4jj8\" (UniqueName: \"kubernetes.io/projected/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-api-access-g4jj8\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.697830 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.697613    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-sys\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.697830 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.697710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.697830 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.697789    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-wtmp\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.697830 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.697819    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-textfile\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.698214 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.697823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-accelerators-collector-config\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.698214 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.697889    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-root\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.698214 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.697922    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4201a324-4bf9-46fc-b1be-c8381f3e9dba-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.698214 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.697952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl4qt\" (UniqueName: \"kubernetes.io/projected/92d09c10-51ac-4231-8195-d0c853887162-kube-api-access-cl4qt\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.698214 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.697979    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92d09c10-51ac-4231-8195-d0c853887162-metrics-client-ca\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.698428 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.698380    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-accelerators-collector-config\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.698500 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.698464    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92d09c10-51ac-4231-8195-d0c853887162-metrics-client-ca\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.698686 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.698661    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4201a324-4bf9-46fc-b1be-c8381f3e9dba-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.698803 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.698785    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-root\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.698960 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.698933    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-wtmp\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.699031 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.698965    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4201a324-4bf9-46fc-b1be-c8381f3e9dba-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.699031 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.698984    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.699031 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.699011    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-sys\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.700350 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.700328    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.700516 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.700495    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-tls\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.700684 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.700668    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.700735 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.700716    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.711840 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.711810    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4jj8\" (UniqueName: \"kubernetes.io/projected/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-api-access-g4jj8\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.712094 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.712078    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl4qt\" (UniqueName: \"kubernetes.io/projected/92d09c10-51ac-4231-8195-d0c853887162-kube-api-access-cl4qt\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.752167 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.752140    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:31:42.762440 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.762418    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:31:42.777172 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:31:42.776834    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d09c10_51ac_4231_8195_d0c853887162.slice/crio-3d8aee8fb5c9641668f90f7de52260d610f8cf81bdeac276d590dfe26f80d536 WatchSource:0}: Error finding container 3d8aee8fb5c9641668f90f7de52260d610f8cf81bdeac276d590dfe26f80d536: Status 404 returned error can't find the container with id 3d8aee8fb5c9641668f90f7de52260d610f8cf81bdeac276d590dfe26f80d536
Oct 30 19:31:42.851204 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:42.851175    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4jpzb" event={"ID":"92d09c10-51ac-4231-8195-d0c853887162","Type":"ContainerStarted","Data":"3d8aee8fb5c9641668f90f7de52260d610f8cf81bdeac276d590dfe26f80d536"}
Oct 30 19:31:43.030545 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.030472    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"]
Oct 30 19:31:43.033712 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:31:43.033687    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4201a324_4bf9_46fc_b1be_c8381f3e9dba.slice/crio-2e97fc75dc8951d5cc89ff0ef02dd775bdcf7d5394b77e9c7a9a05bbd655a3b5 WatchSource:0}: Error finding container 2e97fc75dc8951d5cc89ff0ef02dd775bdcf7d5394b77e9c7a9a05bbd655a3b5: Status 404 returned error can't find the container with id 2e97fc75dc8951d5cc89ff0ef02dd775bdcf7d5394b77e9c7a9a05bbd655a3b5
Oct 30 19:31:43.562961 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.562932    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:31:43.566872 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.566851    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.573770 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.573737    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 19:31:43.574364 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.574021    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 19:31:43.574364 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.574145    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 19:31:43.574364 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.574192    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 19:31:43.574364 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.574265    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 19:31:43.574364 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.574270    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 19:31:43.574364 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.574334    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 19:31:43.575099 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.574369    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 19:31:43.575099 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.574517    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 19:31:43.575099 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.574563    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hwkcq\""
Oct 30 19:31:43.590469 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.590429    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:31:43.605857 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.605818    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.605857 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.605875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.608117 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.605999    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xn9h\" (UniqueName: \"kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-kube-api-access-7xn9h\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.608117 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.606125    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.608117 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.606156    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.608117 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.606241    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.608117 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.606299    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-tls-assets\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.608117 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.606338    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-web-config\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.608117 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.606368    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.608505 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.608135    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-config-volume\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.608505 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.608163    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-config-out\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.608505 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.608183    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.608505 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.608201    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.709654 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.709616    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-config-out\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.709802 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.709662    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.709802 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.709688    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.709802 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.709737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.709974 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.709795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.709974 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.709888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xn9h\" (UniqueName: \"kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-kube-api-access-7xn9h\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.709974 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.709924    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.709974 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.709948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.710186 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.710012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.710186 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.710072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-tls-assets\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.710186 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.710114    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-web-config\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.710186 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.710146    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.710357 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.710202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-config-volume\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.712679 ip-10-0-142-167 kubenswrapper[2576]: E1030 19:31:43.710705    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-trusted-ca-bundle podName:87f098c7-fd9e-4ca8-85ef-412371228817 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:44.210684613 +0000 UTC m=+55.251551377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "87f098c7-fd9e-4ca8-85ef-412371228817") : configmap references non-existent config key: ca-bundle.crt
Oct 30 19:31:43.712679 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.711735    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.712858 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.712716    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.712984 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.712963    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-config-out\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.713591 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.713565    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-config-volume\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.715619 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.715584    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.715944 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.715916    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-tls-assets\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.716339 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.716318    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.716738 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.716716    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-web-config\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.722503 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.722481    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.726515 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.726488    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.731544 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.731519    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.745950 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.745915    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xn9h\" (UniqueName: \"kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-kube-api-access-7xn9h\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:43.855445 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:43.855370    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp" event={"ID":"4201a324-4bf9-46fc-b1be-c8381f3e9dba","Type":"ContainerStarted","Data":"2e97fc75dc8951d5cc89ff0ef02dd775bdcf7d5394b77e9c7a9a05bbd655a3b5"}
Oct 30 19:31:44.219571 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:44.219520    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:44.220528 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:44.220499    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:44.497688 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:44.497658    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:31:44.779158 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:44.779091    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:31:44.801283 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:44.801254    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:31:44.805298 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:31:44.805267    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f098c7_fd9e_4ca8_85ef_412371228817.slice/crio-d7f07af5464a3d7f7b8c28b7ca81681ec36e68037aa347554caf31757796be2c WatchSource:0}: Error finding container d7f07af5464a3d7f7b8c28b7ca81681ec36e68037aa347554caf31757796be2c: Status 404 returned error can't find the container with id d7f07af5464a3d7f7b8c28b7ca81681ec36e68037aa347554caf31757796be2c
Oct 30 19:31:44.860969 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:44.860936    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp" event={"ID":"4201a324-4bf9-46fc-b1be-c8381f3e9dba","Type":"ContainerStarted","Data":"4d7d13606df9006685fef4fca8abb405e079c3a53d4d2114922c82a16ef6f570"}
Oct 30 19:31:44.861185 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:44.860999    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp" event={"ID":"4201a324-4bf9-46fc-b1be-c8381f3e9dba","Type":"ContainerStarted","Data":"400da489926eef46859482ce28dcd6cb1b0f14ba1bd14d4e7cc7128a2903faaf"}
Oct 30 19:31:44.861185 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:44.861028    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp" event={"ID":"4201a324-4bf9-46fc-b1be-c8381f3e9dba","Type":"ContainerStarted","Data":"847734d6c0395c2ffa0f60b3f6e6704bc63f8e5d6bded52e8a1616efd93a232b"}
Oct 30 19:31:44.862347 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:44.862324    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"87f098c7-fd9e-4ca8-85ef-412371228817","Type":"ContainerStarted","Data":"d7f07af5464a3d7f7b8c28b7ca81681ec36e68037aa347554caf31757796be2c"}
Oct 30 19:31:44.863924 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:44.863896    2576 generic.go:356] "Generic (PLEG): container finished" podID="92d09c10-51ac-4231-8195-d0c853887162" containerID="897dbe6ce0ea8ee1149a909c91506e74468871ebec76e706cc39087d87f3612a" exitCode=0
Oct 30 19:31:44.864000 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:44.863938    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4jpzb" event={"ID":"92d09c10-51ac-4231-8195-d0c853887162","Type":"ContainerDied","Data":"897dbe6ce0ea8ee1149a909c91506e74468871ebec76e706cc39087d87f3612a"}
Oct 30 19:31:44.883109 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:44.883028    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp" podStartSLOduration=1.522770899 podStartE2EDuration="2.88301058s" totalImagesPullingTime="1.360239681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:44.881872656 +0000 UTC m=+55.922739422" watchObservedRunningTime="2026-10-30 19:31:44.88301058 +0000 UTC m=+55.923877346"
Oct 30 19:31:45.868906 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:45.868877    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4jpzb" event={"ID":"92d09c10-51ac-4231-8195-d0c853887162","Type":"ContainerStarted","Data":"8579bf190e0cac39bd86c8bb0a76d37d56fe5fb9123b050bbbfef8215a1d5223"}
Oct 30 19:31:45.869285 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:45.868936    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4jpzb" event={"ID":"92d09c10-51ac-4231-8195-d0c853887162","Type":"ContainerStarted","Data":"9cb0edbe55a7dad8eedc7b5d404fba361557956a54f711b7c98f9dda64cc63ca"}
Oct 30 19:31:45.870302 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:45.870277    2576 generic.go:356] "Generic (PLEG): container finished" podID="87f098c7-fd9e-4ca8-85ef-412371228817" containerID="3de25613b3637d81768b30d3e67b2c05d438c51ccf34e834f06ad459f38dbd6c" exitCode=0
Oct 30 19:31:45.870399 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:45.870364    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"87f098c7-fd9e-4ca8-85ef-412371228817","Type":"ContainerDied","Data":"3de25613b3637d81768b30d3e67b2c05d438c51ccf34e834f06ad459f38dbd6c"}
Oct 30 19:31:45.891841 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:45.891790    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4jpzb" podStartSLOduration=2.741661464 podStartE2EDuration="3.891775835s" totalImagesPullingTime="1.063304385s" totalInitContainerRuntime="86.809986ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:45.891181745 +0000 UTC m=+56.932048507" watchObservedRunningTime="2026-10-30 19:31:45.891775835 +0000 UTC m=+56.932642637"
Oct 30 19:31:47.880161 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:47.880130    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"87f098c7-fd9e-4ca8-85ef-412371228817","Type":"ContainerStarted","Data":"5c50947ee1afb5392f692fc6554151edf7227c841ebadc5966b986781edd8fe0"}
Oct 30 19:31:47.880582 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:47.880206    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"87f098c7-fd9e-4ca8-85ef-412371228817","Type":"ContainerStarted","Data":"70a9a42c7f2556f37fcb44d62617400e3daee39e31e4e97e2f40d141a0b1f2ee"}
Oct 30 19:31:47.880582 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:47.880249    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"87f098c7-fd9e-4ca8-85ef-412371228817","Type":"ContainerStarted","Data":"43abd83f9dcccde3a0c795a13ee94cfdbcaeb7f93e482b59abe9819b5fefe9c9"}
Oct 30 19:31:47.880582 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:47.880290    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"87f098c7-fd9e-4ca8-85ef-412371228817","Type":"ContainerStarted","Data":"72126c790b8a7908e6a37fb12dae90c4cf6a449e22e82b63c8a66a064e3a9a6d"}
Oct 30 19:31:47.880582 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:47.880333    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"87f098c7-fd9e-4ca8-85ef-412371228817","Type":"ContainerStarted","Data":"541b595e2bbf504fb8b33ea5d448e2c4a600f864e80eff041795a075edb0e888"}
Oct 30 19:31:48.841708 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:48.841672    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:31:48.887907 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:48.887872    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"87f098c7-fd9e-4ca8-85ef-412371228817","Type":"ContainerStarted","Data":"7448b70599719707de120df46e9e4207a40924bcede75e7e82d7cbadec9c038a"}
Oct 30 19:31:48.924135 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:48.924073    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.802496801 podStartE2EDuration="5.92405462s" totalImagesPullingTime="3.110348426s" totalInitContainerRuntime="11.209393ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:48.923136371 +0000 UTC m=+59.964003135" watchObservedRunningTime="2026-10-30 19:31:48.92405462 +0000 UTC m=+59.964921373"
Oct 30 19:31:55.212878 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.212830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs\") pod \"network-metrics-daemon-xmzrn\" (UID: \"a8eb8c11-ccd0-457f-92bd-c21db1c89e60\") " pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:55.215195 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.215176    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:31:55.225817 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.225782    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs\") pod \"network-metrics-daemon-xmzrn\" (UID: \"a8eb8c11-ccd0-457f-92bd-c21db1c89e60\") " pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:55.313491 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.313457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:55.315690 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.315664    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:31:55.326884 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.326860    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:55.414417 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.414387    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xn78\" (UniqueName: \"kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78\") pod \"network-check-target-hr4sv\" (UID: \"be888c82-964b-42ae-948a-e6e27d276347\") " pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:55.416876 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.416852    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:31:55.427014 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.426982    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:31:55.438061 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.438018    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xn78\" (UniqueName: \"kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78\") pod \"network-check-target-hr4sv\" (UID: \"be888c82-964b-42ae-948a-e6e27d276347\") " pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:55.455185 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.455158    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rpkw7\""
Oct 30 19:31:55.461680 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.461654    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wr97x\""
Oct 30 19:31:55.464029 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.463965    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:31:55.466354 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.466332    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jznq8\""
Oct 30 19:31:55.469424 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.469404    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:31:55.475021 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.475001    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:55.761645 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.761559    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6vm8t"]
Oct 30 19:31:55.765076 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:31:55.765026    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a73dcc_6a78_4344_b0b3_7a8611466a41.slice/crio-aea6acd546a4bb51f2569595c8c3c6001c04c094b87b254989cbe76316a9c0eb WatchSource:0}: Error finding container aea6acd546a4bb51f2569595c8c3c6001c04c094b87b254989cbe76316a9c0eb: Status 404 returned error can't find the container with id aea6acd546a4bb51f2569595c8c3c6001c04c094b87b254989cbe76316a9c0eb
Oct 30 19:31:55.795219 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.795153    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xmzrn"]
Oct 30 19:31:55.801079 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.798111    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hr4sv"]
Oct 30 19:31:55.803470 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:31:55.803441    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8eb8c11_ccd0_457f_92bd_c21db1c89e60.slice/crio-36320ef28ce33f5f0b43cc11e6ce6368151264e568b9b75c0e2b976772f7495d WatchSource:0}: Error finding container 36320ef28ce33f5f0b43cc11e6ce6368151264e568b9b75c0e2b976772f7495d: Status 404 returned error can't find the container with id 36320ef28ce33f5f0b43cc11e6ce6368151264e568b9b75c0e2b976772f7495d
Oct 30 19:31:55.803725 ip-10-0-142-167 kubenswrapper[2576]: W1030 19:31:55.803698    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe888c82_964b_42ae_948a_e6e27d276347.slice/crio-98ec24c765ccebd3f3ba457ae182a13e539380026cd528c65788a82e844adae3 WatchSource:0}: Error finding container 98ec24c765ccebd3f3ba457ae182a13e539380026cd528c65788a82e844adae3: Status 404 returned error can't find the container with id 98ec24c765ccebd3f3ba457ae182a13e539380026cd528c65788a82e844adae3
Oct 30 19:31:55.920628 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.920585    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hr4sv" event={"ID":"be888c82-964b-42ae-948a-e6e27d276347","Type":"ContainerStarted","Data":"98ec24c765ccebd3f3ba457ae182a13e539380026cd528c65788a82e844adae3"}
Oct 30 19:31:55.922641 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.922604    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xmzrn" event={"ID":"a8eb8c11-ccd0-457f-92bd-c21db1c89e60","Type":"ContainerStarted","Data":"36320ef28ce33f5f0b43cc11e6ce6368151264e568b9b75c0e2b976772f7495d"}
Oct 30 19:31:55.923878 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:55.923846    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6vm8t" event={"ID":"b7a73dcc-6a78-4344-b0b3-7a8611466a41","Type":"ContainerStarted","Data":"aea6acd546a4bb51f2569595c8c3c6001c04c094b87b254989cbe76316a9c0eb"}
Oct 30 19:31:57.936368 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:57.936255    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xmzrn" event={"ID":"a8eb8c11-ccd0-457f-92bd-c21db1c89e60","Type":"ContainerStarted","Data":"fddd2ee42225fdfc17317cbd87756bf60f9ee1560b3f036586b5368c4cfd8fba"}
Oct 30 19:31:57.936368 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:57.936330    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xmzrn" event={"ID":"a8eb8c11-ccd0-457f-92bd-c21db1c89e60","Type":"ContainerStarted","Data":"3ffaad0253887000a4646b26e98146dca9e9f97508e42ecefa70f4de5ccea41a"}
Oct 30 19:31:59.946627 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:59.946586    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hr4sv" event={"ID":"be888c82-964b-42ae-948a-e6e27d276347","Type":"ContainerStarted","Data":"55979f69bbea5e33a6335d260c9efd92f63b712860476ac8be45ebd37f73b29b"}
Oct 30 19:31:59.947085 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:59.946758    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:31:59.969393 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:59.969325    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hr4sv" podStartSLOduration=67.478196008 podStartE2EDuration="1m10.969304977s" totalImagesPullingTime="3.491108969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:59.96819218 +0000 UTC m=+71.009058942" watchObservedRunningTime="2026-10-30 19:31:59.969304977 +0000 UTC m=+71.010171742"
Oct 30 19:31:59.970497 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:31:59.970444    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xmzrn" podStartSLOduration=69.438453352 podStartE2EDuration="1m10.970428531s" totalImagesPullingTime="1.531975179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:57.955739552 +0000 UTC m=+68.996606314" watchObservedRunningTime="2026-10-30 19:31:59.970428531 +0000 UTC m=+71.011295296"
Oct 30 19:32:00.952591 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:32:00.952547    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6vm8t" event={"ID":"b7a73dcc-6a78-4344-b0b3-7a8611466a41","Type":"ContainerStarted","Data":"94a5d6a02c173774281c41ce105d86cfba2fc3858b16e294039483ac2e95c009"}
Oct 30 19:32:00.973242 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:32:00.973190    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6vm8t" podStartSLOduration=66.98671468 podStartE2EDuration="1m11.973175725s" totalImagesPullingTime="4.986461045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:00.971934886 +0000 UTC m=+72.012801650" watchObservedRunningTime="2026-10-30 19:32:00.973175725 +0000 UTC m=+72.014042488"
Oct 30 19:32:01.013359 ip-10-0-142-167 kubenswrapper[2576]: I1030 19:32:01.013257    2576 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:32:01.013430 ip-10-0-142-167 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 19:32:01.030426 ip-10-0-142-167 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 19:32:01.030603 ip-10-0-142-167 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 19:32:01.030661 ip-10-0-142-167 systemd[1]: kubelet.service: Consumed 3.981s CPU time.
Oct 30 19:32:01.049582 ip-10-0-142-167 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:32:01.113440 ip-10-0-142-167 kubenswrapper[6030]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:32:01.113440 ip-10-0-142-167 kubenswrapper[6030]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:32:01.113440 ip-10-0-142-167 kubenswrapper[6030]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:32:01.113440 ip-10-0-142-167 kubenswrapper[6030]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:32:01.115587 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115499    6030 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:32:01.115587 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115587    6030 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115591    6030 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115594    6030 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115597    6030 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115600    6030 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115609    6030 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115614    6030 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115617    6030 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115620    6030 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115623    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115625    6030 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115628    6030 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115631    6030 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115633    6030 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115636    6030 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115638    6030 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115641    6030 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115643    6030 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115646    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:32:01.115654 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115648    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115651    6030 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115653    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115656    6030 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115659    6030 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115661    6030 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115664    6030 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115667    6030 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115670    6030 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115672    6030 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115674    6030 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115677    6030 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115680    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115682    6030 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115685    6030 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115687    6030 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115691    6030 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115695    6030 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115698    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115700    6030 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:32:01.116181 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115703    6030 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115705    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115707    6030 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115710    6030 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115712    6030 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115715    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115717    6030 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115720    6030 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115722    6030 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115724    6030 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115727    6030 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115730    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115732    6030 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115735    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115738    6030 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115741    6030 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115743    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115745    6030 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115748    6030 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:32:01.116777 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115751    6030 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115753    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115755    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115758    6030 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115761    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115763    6030 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115766    6030 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115769    6030 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115771    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115774    6030 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115776    6030 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115779    6030 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115781    6030 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115783    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115786    6030 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115788    6030 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115793    6030 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115795    6030 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115798    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:32:01.117272 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115801    6030 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115803    6030 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115806    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115809    6030 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115811    6030 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115815    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115818    6030 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115820    6030 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115823    6030 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115825    6030 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115828    6030 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115830    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115833    6030 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115835    6030 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115838    6030 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115840    6030 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115842    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115845    6030 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115848    6030 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:32:01.117770 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115851    6030 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115854    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115857    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115860    6030 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115862    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115865    6030 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115867    6030 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115869    6030 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.115873    6030 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116685    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116696    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116699    6030 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116703    6030 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116705    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116708    6030 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116711    6030 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116714    6030 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116716    6030 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116719    6030 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116721    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:32:01.118276 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116724    6030 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116727    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116729    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116732    6030 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116734    6030 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116737    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116739    6030 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116741    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116744    6030 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116746    6030 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116749    6030 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116751    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116754    6030 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116756    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116760    6030 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116762    6030 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116765    6030 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116767    6030 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116770    6030 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116772    6030 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:32:01.118776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116774    6030 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116777    6030 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116781    6030 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116784    6030 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116787    6030 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116789    6030 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116792    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116794    6030 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116797    6030 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116800    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116802    6030 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116805    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116807    6030 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116810    6030 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116812    6030 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116814    6030 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116817    6030 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116819    6030 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116822    6030 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116825    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:32:01.119285 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116827    6030 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116830    6030 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116832    6030 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116835    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116837    6030 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116840    6030 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116842    6030 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116845    6030 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116848    6030 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116850    6030 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116853    6030 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116855    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116858    6030 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116860    6030 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116864    6030 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116866    6030 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116870    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116872    6030 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116875    6030 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116877    6030 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:32:01.119771 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116881    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116883    6030 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116886    6030 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116888    6030 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116891    6030 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116893    6030 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116896    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116898    6030 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116900    6030 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116903    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116905    6030 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116908    6030 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116910    6030 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116913    6030 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116915    6030 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116918    6030 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116920    6030 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116923    6030 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116925    6030 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:32:01.120298 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116929    6030 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116934    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116937    6030 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116939    6030 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116942    6030 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116945    6030 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116947    6030 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116951    6030 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116954    6030 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116956    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116959    6030 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116961    6030 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116966    6030 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116970    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116973    6030 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.116976    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117464    6030 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117470    6030 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117473    6030 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:32:01.120768 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117476    6030 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117478    6030 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117481    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117483    6030 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117486    6030 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117488    6030 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117491    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117493    6030 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117496    6030 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117498    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117501    6030 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117504    6030 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117507    6030 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117509    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117512    6030 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117515    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117518    6030 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117520    6030 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117523    6030 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:32:01.121273 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117525    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117528    6030 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117531    6030 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117533    6030 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117536    6030 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117538    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117540    6030 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117543    6030 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117546    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117548    6030 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117550    6030 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117554    6030 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117558    6030 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117561    6030 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117563    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117566    6030 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117568    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117571    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117573    6030 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:32:01.122014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117575    6030 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117578    6030 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117580    6030 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117583    6030 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117585    6030 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117588    6030 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117590    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117593    6030 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117595    6030 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117598    6030 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117601    6030 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117603    6030 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117606    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117608    6030 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117611    6030 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117613    6030 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117615    6030 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117618    6030 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117620    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117623    6030 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:32:01.122869 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117625    6030 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117628    6030 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117631    6030 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117633    6030 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117637    6030 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117639    6030 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117643    6030 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117646    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117649    6030 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117652    6030 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117654    6030 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117657    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117659    6030 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117662    6030 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117664    6030 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117667    6030 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117669    6030 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117672    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117674    6030 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117677    6030 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:32:01.123516 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117679    6030 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117683    6030 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117686    6030 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117688    6030 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117690    6030 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117693    6030 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117695    6030 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117698    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117700    6030 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117702    6030 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117705    6030 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117707    6030 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117710    6030 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117712    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117715    6030 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117718    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117720    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117723    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117725    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117727    6030 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:32:01.124014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117730    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117733    6030 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117735    6030 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117738    6030 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.117741    6030 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117834    6030 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117841    6030 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117848    6030 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117853    6030 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117857    6030 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117861    6030 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117877    6030 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117882    6030 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117885    6030 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117888    6030 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117892    6030 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117895    6030 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117898    6030 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117901    6030 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117904    6030 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117909    6030 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117912    6030 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117915    6030 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:32:01.124525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117920    6030 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117922    6030 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117925    6030 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117929    6030 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117932    6030 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117935    6030 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117938    6030 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117941    6030 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117945    6030 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117948    6030 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117951    6030 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117954    6030 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117957    6030 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117960    6030 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117964    6030 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117967    6030 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117970    6030 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117973    6030 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117976    6030 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117979    6030 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117984    6030 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117987    6030 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117990    6030 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117993    6030 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:32:01.125141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.117996    6030 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118000    6030 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118003    6030 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118006    6030 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118009    6030 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118013    6030 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118016    6030 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118019    6030 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118022    6030 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118025    6030 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118028    6030 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118031    6030 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118035    6030 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118038    6030 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118057    6030 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118060    6030 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118064    6030 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118067    6030 flags.go:64] FLAG: --help="false"
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118070    6030 flags.go:64] FLAG: --hostname-override="ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118074    6030 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118077    6030 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118079    6030 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118083    6030 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:32:01.125767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118086    6030 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118089    6030 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118091    6030 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118094    6030 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118097    6030 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118100    6030 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118104    6030 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118106    6030 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118109    6030 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118112    6030 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118116    6030 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118119    6030 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118122    6030 flags.go:64] FLAG: --lock-file=""
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118125    6030 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118129    6030 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118132    6030 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118137    6030 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118140    6030 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118143    6030 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118146    6030 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118149    6030 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118152    6030 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118155    6030 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118157    6030 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118162    6030 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:32:01.126342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118166    6030 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118170    6030 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118172    6030 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118175    6030 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118178    6030 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118181    6030 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118184    6030 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118187    6030 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118190    6030 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118196    6030 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118199    6030 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118202    6030 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118205    6030 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118208    6030 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118211    6030 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118214    6030 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118217    6030 flags.go:64] FLAG: --port="10250"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118219    6030 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118222    6030 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0886917c7464b7813"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118226    6030 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118229    6030 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118233    6030 flags.go:64] FLAG: --register-node="true"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118236    6030 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118240    6030 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118242    6030 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118246    6030 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:32:01.127061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118249    6030 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118252    6030 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118255    6030 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118258    6030 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118262    6030 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118264    6030 flags.go:64] FLAG: --runonce="false"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118269    6030 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118273    6030 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118276    6030 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118279    6030 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118282    6030 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118285    6030 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118288    6030 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118291    6030 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118294    6030 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118297    6030 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118300    6030 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118303    6030 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118306    6030 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118308    6030 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118311    6030 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118317    6030 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118320    6030 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118323    6030 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118327    6030 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:32:01.127715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118329    6030 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118333    6030 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118337    6030 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118340    6030 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118343    6030 flags.go:64] FLAG: --v="2"
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118347    6030 flags.go:64] FLAG: --version="false"
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118351    6030 flags.go:64] FLAG: --vmodule=""
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118355    6030 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118358    6030 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118490    6030 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118493    6030 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118496    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118499    6030 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118503    6030 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118505    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118508    6030 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118511    6030 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118513    6030 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118516    6030 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118519    6030 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118521    6030 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:32:01.128408 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118524    6030 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118526    6030 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118529    6030 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118531    6030 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118534    6030 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118536    6030 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118538    6030 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118541    6030 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118543    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118546    6030 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118548    6030 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118551    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118553    6030 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118557    6030 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118560    6030 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118562    6030 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118565    6030 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118567    6030 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118570    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118572    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:32:01.129013 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118574    6030 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118577    6030 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118580    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118582    6030 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118586    6030 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118588    6030 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118591    6030 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118593    6030 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118596    6030 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118599    6030 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118601    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118604    6030 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118606    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118609    6030 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118611    6030 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118614    6030 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118616    6030 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118620    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118622    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:32:01.129593 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118625    6030 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118627    6030 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118629    6030 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118632    6030 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118634    6030 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118644    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118649    6030 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118652    6030 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118655    6030 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118658    6030 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118660    6030 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118663    6030 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118666    6030 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118668    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118671    6030 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118673    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118675    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118679    6030 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118681    6030 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118684    6030 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:32:01.130248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118687    6030 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118689    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118692    6030 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118694    6030 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118697    6030 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118699    6030 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118702    6030 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118705    6030 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118707    6030 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118711    6030 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118715    6030 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118718    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118721    6030 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118723    6030 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118725    6030 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118728    6030 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118730    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118733    6030 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118737    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:32:01.130739 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118740    6030 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:32:01.131248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118743    6030 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:32:01.131248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118745    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:32:01.131248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118748    6030 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:32:01.131248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118750    6030 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:32:01.131248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118752    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:32:01.131248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118755    6030 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:32:01.131248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118757    6030 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:32:01.131248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118760    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:32:01.131248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118762    6030 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:32:01.131248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118766    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:32:01.131248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118768    6030 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:32:01.131248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118771    6030 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:32:01.131248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118773    6030 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:32:01.131248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118776    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:32:01.131248 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118779    6030 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:32:01.131248 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.118806    6030 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118848    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118853    6030 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118857    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118860    6030 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118863    6030 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118866    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118869    6030 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118871    6030 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118874    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118877    6030 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118879    6030 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118882    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118884    6030 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118887    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118890    6030 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118893    6030 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118895    6030 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118899    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:32:01.131838 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118903    6030 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118907    6030 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118911    6030 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118913    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118916    6030 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118918    6030 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118921    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118923    6030 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118928    6030 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118931    6030 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118933    6030 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118936    6030 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118938    6030 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118941    6030 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118943    6030 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118945    6030 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118948    6030 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118950    6030 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118953    6030 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118955    6030 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:32:01.132644 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118958    6030 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118960    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118962    6030 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118965    6030 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118967    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118970    6030 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118972    6030 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118975    6030 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118979    6030 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118982    6030 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118984    6030 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118987    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118990    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118993    6030 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118995    6030 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.118997    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119000    6030 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119002    6030 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119005    6030 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119007    6030 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:32:01.133169 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119011    6030 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119013    6030 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119017    6030 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119020    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119022    6030 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119025    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119027    6030 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119030    6030 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119032    6030 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119035    6030 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119038    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119053    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119056    6030 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119058    6030 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119061    6030 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119063    6030 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119066    6030 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119068    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119071    6030 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:32:01.133646 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119073    6030 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119077    6030 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119080    6030 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119082    6030 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119085    6030 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119087    6030 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119090    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119093    6030 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119096    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119098    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119101    6030 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119103    6030 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119106    6030 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119112    6030 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119114    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119117    6030 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119119    6030 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119122    6030 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119124    6030 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119127    6030 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:32:01.134124 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119129    6030 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119132    6030 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119136    6030 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119140    6030 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119143    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119146    6030 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119149    6030 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119152    6030 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.119154    6030 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.124170    6030 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.124189    6030 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.124204    6030 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.124210    6030 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124295    6030 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124301    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124304    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124307    6030 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124310    6030 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124313    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:32:01.134634 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124316    6030 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124319    6030 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124321    6030 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124324    6030 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124326    6030 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124329    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124332    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124335    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124337    6030 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124340    6030 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124343    6030 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124346    6030 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124349    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124351    6030 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124354    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124357    6030 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124359    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124362    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124365    6030 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:32:01.135159 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124367    6030 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124370    6030 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124372    6030 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124375    6030 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124378    6030 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124380    6030 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124383    6030 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124385    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124388    6030 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124390    6030 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124393    6030 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124395    6030 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124398    6030 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124400    6030 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124403    6030 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124405    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124407    6030 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124410    6030 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124412    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124415    6030 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:32:01.135773 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124417    6030 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124421    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124423    6030 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124426    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124429    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124431    6030 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124434    6030 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124436    6030 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124439    6030 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124441    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124443    6030 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124446    6030 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124449    6030 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124451    6030 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124454    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124456    6030 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124459    6030 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124461    6030 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124464    6030 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:32:01.136475 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124466    6030 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124470    6030 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124474    6030 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124477    6030 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124479    6030 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124482    6030 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124484    6030 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124487    6030 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124489    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124492    6030 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124495    6030 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124497    6030 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124499    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124502    6030 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124504    6030 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124507    6030 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124511    6030 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124514    6030 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124517    6030 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:32:01.137170 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124519    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124522    6030 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124524    6030 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124527    6030 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124529    6030 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124532    6030 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124536    6030 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124539    6030 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124543    6030 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124545    6030 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124549    6030 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124552    6030 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124555    6030 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124557    6030 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124559    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124562    6030 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124564    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124567    6030 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124569    6030 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124571    6030 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:32:01.137691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124574    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:32:01.138302 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124576    6030 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:32:01.138302 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124579    6030 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:32:01.138302 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.124607    6030 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:32:01.138302 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124751    6030 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:32:01.138302 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124756    6030 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:32:01.138302 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124759    6030 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:32:01.138302 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124761    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:32:01.138302 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124764    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:32:01.138302 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124767    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:32:01.138302 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124770    6030 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:32:01.138302 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124772    6030 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:32:01.138302 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124775    6030 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:32:01.138302 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124778    6030 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:32:01.138302 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124780    6030 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:32:01.138302 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124782    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:32:01.138302 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124785    6030 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124787    6030 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124790    6030 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124792    6030 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124795    6030 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124797    6030 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124799    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124802    6030 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124804    6030 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124807    6030 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124809    6030 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124812    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124814    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124817    6030 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124819    6030 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124822    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124824    6030 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124827    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124829    6030 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124831    6030 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:32:01.138811 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124834    6030 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124836    6030 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124839    6030 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124841    6030 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124843    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124846    6030 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124850    6030 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124853    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124855    6030 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124859    6030 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124862    6030 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124866    6030 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124870    6030 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124873    6030 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124876    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124878    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124881    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124883    6030 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124886    6030 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:32:01.139465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124888    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124891    6030 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124893    6030 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124896    6030 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124898    6030 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124901    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124903    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124906    6030 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124908    6030 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124911    6030 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124913    6030 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124916    6030 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124918    6030 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124921    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124923    6030 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124926    6030 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124928    6030 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124931    6030 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124933    6030 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124936    6030 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:32:01.140097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124939    6030 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124941    6030 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124944    6030 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124946    6030 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124949    6030 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124951    6030 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124954    6030 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124956    6030 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124959    6030 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124962    6030 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124964    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124967    6030 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124969    6030 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124972    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124974    6030 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124977    6030 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124979    6030 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124981    6030 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124984    6030 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:32:01.140879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124986    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:32:01.141577 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124989    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:32:01.141577 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124991    6030 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:32:01.141577 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124993    6030 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:32:01.141577 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124996    6030 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:32:01.141577 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.124998    6030 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:32:01.141577 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125001    6030 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:32:01.141577 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125003    6030 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:32:01.141577 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125005    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:32:01.141577 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125008    6030 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:32:01.141577 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125010    6030 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:32:01.141577 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125013    6030 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:32:01.141577 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125015    6030 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:32:01.141577 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125017    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:32:01.141577 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125021    6030 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:32:01.141577 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125024    6030 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:32:01.141577 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.125076    6030 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125110    6030 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125114    6030 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125117    6030 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125120    6030 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125122    6030 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125125    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125127    6030 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125129    6030 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125132    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125135    6030 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125138    6030 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125142    6030 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125146    6030 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125149    6030 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125152    6030 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125155    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125158    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125160    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:32:01.141971 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125162    6030 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125165    6030 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125167    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125170    6030 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125172    6030 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125175    6030 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125177    6030 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125180    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125183    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125186    6030 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125188    6030 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125191    6030 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125193    6030 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125196    6030 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125198    6030 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125202    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125204    6030 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125207    6030 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125209    6030 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125212    6030 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:32:01.142592 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125214    6030 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125217    6030 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125219    6030 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125221    6030 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125224    6030 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125226    6030 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125229    6030 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125231    6030 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125233    6030 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125236    6030 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125238    6030 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125241    6030 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125243    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125246    6030 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125248    6030 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125251    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125253    6030 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125255    6030 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125258    6030 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125260    6030 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:32:01.143115 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125263    6030 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125265    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125267    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125270    6030 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125272    6030 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125275    6030 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125278    6030 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125280    6030 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125282    6030 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125285    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125287    6030 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125289    6030 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125292    6030 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125294    6030 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125297    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125299    6030 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125302    6030 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125304    6030 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125309    6030 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125312    6030 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:32:01.143670 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125314    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125317    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125319    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125322    6030 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125325    6030 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125327    6030 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125329    6030 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125332    6030 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125335    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125337    6030 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125340    6030 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125344    6030 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125346    6030 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125349    6030 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125351    6030 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125353    6030 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125356    6030 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125358    6030 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125362    6030 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:32:01.144350 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125364    6030 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:32:01.145162 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125367    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:32:01.145162 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125369    6030 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:32:01.145162 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125372    6030 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:32:01.145162 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125374    6030 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:32:01.145162 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125376    6030 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:32:01.145162 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125380    6030 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:32:01.145162 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125383    6030 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:32:01.145162 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:01.125385    6030 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:32:01.145162 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.125443    6030 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:32:01.145162 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.126417    6030 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 19:32:01.145162 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.126473    6030 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 19:32:01.145162 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.126651    6030 server.go:1014] "Starting client certificate rotation"
Oct 30 19:32:01.145162 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.126795    6030 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:32:01.145162 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.126859    6030 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:25:49 +0000 UTC" deadline="2028-06-01 02:43:07.984652178 +0000 UTC"
Oct 30 19:32:01.145162 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.126898    6030 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13903h11m6.857757599s"
Oct 30 19:32:01.145162 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.127124    6030 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:32:01.145955 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.128199    6030 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:32:01.145955 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.129427    6030 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:32:01.145955 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.131328    6030 log.go:25] "Validated CRI v1 image API"
Oct 30 19:32:01.145955 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.132121    6030 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:32:01.145955 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.134098    6030 fs.go:135] Filesystem UUIDs: map[51d1ecf5-de09-4989-b83d-25c6833076d4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ae03064f-cef8-4f5f-a32d-09ae7eca4d98:/dev/nvme0n1p3]
Oct 30 19:32:01.146356 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.134121    6030 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09163586b4ffabebea3176275992c748d561840fdc4417388c063f201ca008ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09163586b4ffabebea3176275992c748d561840fdc4417388c063f201ca008ac/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e40c5103d22a472d26a9109f4f4897bb9cbb027fa6bbd4cc492602e19d223b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e40c5103d22a472d26a9109f4f4897bb9cbb027fa6bbd4cc492602e19d223b2/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19eebbf5810a27b2a6adafe2b781d5d58f8a50f013fb9ac24422251ae6569f1e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19eebbf5810a27b2a6adafe2b781d5d58f8a50f013fb9ac24422251ae6569f1e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e97fc75dc8951d5cc89ff0ef02dd775bdcf7d5394b77e9c7a9a05bbd655a3b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e97fc75dc8951d5cc89ff0ef02dd775bdcf7d5394b77e9c7a9a05bbd655a3b5/userdata/shm major:0 minor:274 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36320ef28ce33f5f0b43cc11e6ce6368151264e568b9b75c0e2b976772f7495d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36320ef28ce33f5f0b43cc11e6ce6368151264e568b9b75c0e2b976772f7495d/userdata/shm major:0 minor:358 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b5fcf354f1de5ae721c7e59bdbc57b009d0fcd8dffc411c85af9784f33c2318/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b5fcf354f1de5ae721c7e59bdbc57b009d0fcd8dffc411c85af9784f33c2318/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d8aee8fb5c9641668f90f7de52260d610f8cf81bdeac276d590dfe26f80d536/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d8aee8fb5c9641668f90f7de52260d610f8cf81bdeac276d590dfe26f80d536/userdata/shm major:0 minor:277 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ea2b374f029306389eb06c1bedfa13941608fc7e0fe310460a8d3b094709229/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ea2b374f029306389eb06c1bedfa13941608fc7e0fe310460a8d3b094709229/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d2ce302416f14e67ff465ce723e3d0a21e9f1859dd8e1b26044efe09af53447/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d2ce302416f14e67ff465ce723e3d0a21e9f1859dd8e1b26044efe09af53447/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55ec1086b671b271d3dfa310dedfc2c6b40729d72cba4ca9141a1a58756155cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55ec1086b671b271d3dfa310dedfc2c6b40729d72cba4ca9141a1a58756155cd/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a1a321b62a9b9de7d0b59b69e9430c2a74a4f6b70b2d28b081282567e2300b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a1a321b62a9b9de7d0b59b69e9430c2a74a4f6b70b2d28b081282567e2300b4/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70833a700389847c63f703c4b20bba947f3be67b4805ea677d0a1b085ec704dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70833a700389847c63f703c4b20bba947f3be67b4805ea677d0a1b085ec704dd/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98ec24c765ccebd3f3ba457ae182a13e539380026cd528c65788a82e844adae3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98ec24c765ccebd3f3ba457ae182a13e539380026cd528c65788a82e844adae3/userdata/shm major:0 minor:360 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99a447cd455cca759581ade9ee8a552300faff916254f55447b1a5ee5c6461a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99a447cd455cca759581ade9ee8a552300faff916254f55447b1a5ee5c6461a3/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aea6acd546a4bb51f2569595c8c3c6001c04c094b87b254989cbe76316a9c0eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aea6acd546a4bb51f2569595c8c3c6001c04c094b87b254989cbe76316a9c0eb/userdata/shm major:0 minor:356 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7f07af5464a3d7f7b8c28b7ca81681ec36e68037aa347554caf31757796be2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7f07af5464a3d7f7b8c28b7ca81681ec36e68037aa347554caf31757796be2c/userdata/shm major:0 minor:300 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/deae8580d1249abde68c8681ce29ed814663d86a100c519e5bc3ca68198b88ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/deae8580d1249abde68c8681ce29ed814663d86a100c519e5bc3ca68198b88ed/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6ccdc6ef5808392e98132ecb9055e6479ae1eab7bf7ad98ee84375a6adbb6a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6ccdc6ef5808392e98132ecb9055e6479ae1eab7bf7ad98ee84375a6adbb6a5/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f59a3d2fa1f4bdde3a8fe04e2021f2bfc6b47b7e33ecd49e248c1665a784d27b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f59a3d2fa1f4bdde3a8fe04e2021f2bfc6b47b7e33ecd49e248c1665a784d27b/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe2bb906042c92e3f81991602e039cca05ede3fc631204df00ceba6e679577b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe2bb906042c92e3f81991602e039cca05ede3fc631204df00ceba6e679577b5/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fef01e5-9be4-4368-8f1e-3f73d7d6c60d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0fef01e5-9be4-4368-8f1e-3f73d7d6c60d/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35333d53-b864-432d-af6d-b1baec362af6/volumes/kubernetes.io~projected/kube-api-access-mbwlx:{mountpoint:/var/lib/kubelet/pods/35333d53-b864-432d-af6d-b1baec362af6/volumes/kubernetes.io~projected/kube-api-access-mbwlx major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4201a324-4bf9-46fc-b1be-c8381f3e9dba/volumes/kubernetes.io~projected/kube-api-access-g4jj8:{mountpoint:/var/lib/kubelet/pods/4201a324-4bf9-46fc-b1be-c8381f3e9dba/volumes/kubernetes.io~projected/kube-api-access-g4jj8 major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4201a324-4bf9-46fc-b1be-c8381f3e9dba/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/4201a324-4bf9-46fc-b1be-c8381f3e9dba/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4201a324-4bf9-46fc-b1be-c8381f3e9dba/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/4201a324-4bf9-46fc-b1be-c8381f3e9dba/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5aad5010-0827-4918-85d9-47e337a22774/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/5aad5010-0827-4918-85d9-47e337a22774/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5aad5010-0827-4918-85d9-47e337a22774/volumes/kubernetes.io~projected/kube-api-access-84w8n:{mountpoint:/var/lib/kubelet/pods/5aad5010-0827-4918-85d9-47e337a22774/volumes/kubernetes.io~projected/kube-api-access-84w8n major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5aad5010-0827-4918-85d9-47e337a22774/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/5aad5010-0827-4918-85d9-47e337a22774/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6035c60e-c58e-455e-a61f-19eb599272af/volumes/kubernetes.io~projected/kube-api-access-2q94g:{mountpoint:/var/lib/kubelet/pods/6035c60e-c58e-455e-a61f-19eb599272af/volumes/kubernetes.io~projected/kube-api-access-2q94g major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volume-subpaths/web-config/alertmanager/9 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volume-subpaths/web-config/config-reloader/6 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~empty-dir/config-out major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~projected/kube-api-access-7xn9h:{mountpoint:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~projected/kube-api-access-7xn9h major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~projected/tls-assets major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/config-volume major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/web-config major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92d09c10-51ac-4231-8195-d0c853887162/volumes/kubernetes.io~projected/kube-api-access-cl4qt:{mountpoint:/var/lib/kubelet/pods/92d09c10-51ac-4231-8195-d0c853887162/volumes/kubernetes.io~projected/kube-api-access-cl4qt major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92d09c10-51ac-4231-8195-d0c853887162/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/92d09c10-51ac-4231-8195-d0c853887162/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92d09c10-51ac-4231-8195-d0c853887162/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/92d09c10-51ac-4231-8195-d0c853887162/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a675f8c3-6d56-485d-8c87-438d66ebdc1e/volumes/kubernetes.io~projected/kube-api-access-xqggx:{mountpoint:/var/lib/kubelet/pods/a675f8c3-6d56-485d-8c87-438d66ebdc1e/volumes/kubernetes.io~projected/kube-api-access-xqggx major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a675f8c3-6d56-485d-8c87-438d66ebdc1e/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/a675f8c3-6d56-485d-8c87-438d66ebdc1e/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8eb8c11-ccd0-457f-92bd-c21db1c89e60/volumes/kubernetes.io~projected/kube-api-access-7c2bb:{mountpoint:/var/lib/kubelet/pods/a8eb8c11-ccd0-457f-92bd-c21db1c89e60/volumes/kubernetes.io~projected/kube-api-access-7c2bb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8eb8c11-ccd0-457f-92bd-c21db1c89e60/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/a8eb8c11-ccd0-457f-92bd-c21db1c89e60/volumes/kubernetes.io~secret/metrics-certs major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b16dc00a-c3b0-4767-a468-6d204ba44e55/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b16dc00a-c3b0-4767-a468-6d204ba44e55/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b16dc00a-c3b0-4767-a468-6d204ba44e55/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b16dc00a-c3b0-4767-a468-6d204ba44e55/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b16dc00a-c3b0-4767-a468-6d204ba44e55/volumes/kubernetes.io~projected/kube-api-access-7bmxb:{mountpoint:/var/lib/kubelet/pods/b16dc00a-c3b0-4767-a468-6d204ba44e55/volumes/kubernetes.io~projected/kube-api-access-7bmxb major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7a73dcc-6a78-4344-b0b3-7a8611466a41/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b7a73dcc-6a78-4344-b0b3-7a8611466a41/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be888c82-964b-42ae-948a-e6e27d276347/volumes/kubernetes.io~projected/kube-api-access-5xn78:{mountpoint:/var/lib/kubelet/pods/be888c82-964b-42ae-948a-e6e27d276347/volumes/kubernetes.io~projected/kube-api-access-5xn78 major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cdfe62e2-9594-4c48-a3bc-2dce9df78b56/volumes/kubernetes.io~projected/kube-api-access-4dgf7:{mountpoint:/var/lib/kubelet/pods/cdfe62e2-9594-4c48-a3bc-2dce9df78b56/volumes/kubernetes.io~projected/kube-api-access-4dgf7 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cedda70a-9baa-4592-818a-63092c854c8e/volumes/kubernetes.io~projected/kube-api-access-4t74h:{mountpoint:/var/lib/kubelet/pods/cedda70a-9baa-4592-818a-63092c854c8e/volumes/kubernetes.io~projected/kube-api-access-4t74h major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cedda70a-9baa-4592-818a-63092c854c8e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/cedda70a-9baa-4592-818a-63092c854c8e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfda46fc-34fd-461d-a2bf-d4ec19596ed2/volumes/kubernetes.io~projected/kube-api-access-8vpzr:{mountpoint:/var/lib/kubelet/pods/cfda46fc-34fd-461d-a2bf-d4ec19596ed2/volumes/kubernetes.io~projected/kube-api-access-8vpzr major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfda46fc-34fd-461d-a2bf-d4ec19596ed2/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/cfda46fc-34fd-461d-a2bf-d4ec19596ed2/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d432cc60-02df-4a8d-94d4-93aafac213f2/volumes/kubernetes.io~projected/kube-api-access-lk9b4:{mountpoint:/var/lib/kubelet/pods/d432cc60-02df-4a8d-94d4-93aafac213f2/volumes/kubernetes.io~projected/kube-api-access-lk9b4 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee3c157f-683e-415d-b856-74da8b4fe669/volumes/kubernetes.io~projected/kube-api-access-kcfwl:{mountpoint:/var/lib/kubelet/pods/ee3c157f-683e-415d-b856-74da8b4fe669/volumes/kubernetes.io~projected/kube-api-access-kcfwl major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef93f664-f7a4-42b9-bb9c-a10c519e9685/volumes/kubernetes.io~projected/kube-api-access-bb6r7:{mountpoint:/var/lib/kubelet/pods/ef93f664-f7a4-42b9-bb9c-a10c519e9685/volumes/kubernetes.io~projected/kube-api-access-bb6r7 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6b01178-0262-47a5-81bb-d715f5cf4380/volumes/kubernetes.io~projected/kube-api-access-bvm7j:{mountpoint:/var/lib/kubelet/pods/f6b01178-0262-47a5-81bb-d715f5cf4380/volumes/kubernetes.io~projected/kube-api-access-bvm7j major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6b01178-0262-47a5-81bb-d715f5cf4380/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f6b01178-0262-47a5-81bb-d715f5cf4380/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6b01178-0262-47a5-81bb-d715f5cf4380/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/f6b01178-0262-47a5-81bb-d715f5cf4380/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:229 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/2d80fe0cbc4a2b27bd17bd6d455f0d09af45de69b426d2a1d25fbf484864734a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/edd062e86fc9083e994227561bef26ecaefb4bff4ef17a5b94c27b4c1cf0dcd6/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/085ff94b47570abd95c3c4dc2ff35a4bacdb7d148c55dbc3cfb87d1464518ac9/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/c90a39c8c7546273e376f28fad8cb4a6ac2e3f287e5b02b2cda2e7fbef1c3b30/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/40f44bb786d5b534cf8cefe29ef860f2c1a782c3d380acb48d680ca03d549600/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0b4c35bc0cad1b20a989fcfdbde688f38b5d65052e99e604c229a956207b01cc/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/cc2a38c36aea7f2f6ed59ae415439707af049e3cc00a8bb99fd2074643a3f8d1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d46c412ed1e7dfcb2d27071ef4bd4387424ac3186b50372f1b4376ae0246652c/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ffd22338d3ba76342d7ce6e69685bc07dea4b2688d64cdfd24ebfd63c34eea46/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/0c147f5db00729a991238beceb3a0b13f3fc6e505987e2c9afbe97fc5b12410e/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/f0f20026161656db472351855aa6106f320fe1c93bfd3c522ecfbfbd7f327d0a/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/0cbcb42aa967a69abc3850bc0d8ec1b1c12787211fe452df63bcd8117146219d/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e25ee37e85c9d2169b8c5a3f4a25ff7cc6cb12089d25bc00936550199ffc7adf/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/250d2ef163ace6b023416a8bd402c92d9df88a24a152584bb58b6b0b51d314e3/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/b0c6bf1498bb6cf7638fb06caf9f61b06b5813547109dd2181006c3d55a6af2a/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9bbca787422365dbace2ed6428f1a935970b9aa49356ca105c502b01310c78d2/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/7bf933dd1b26d9c166166a0c6d2f4c83ee95569a28f973a8271e2aa3b2c02fe5/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/37d5f8642d3b0eb9b3c87ae62128c01c9ce40255be06cdc7489d2de9eda35b9f/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/17e659002777fd4ba2963bc0b7578c9250ff6b420a0c61979300c0723327641a/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0d60fbfcf46febde086116c5a64bc1d5573139599377bc2b91be7340e2176def/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/630bbe5a772ef4be983f6a52701f3d1156141c44a92d82bcc960244f82fdf760/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/1e883f090e9dacd16efebb568d509d4cbf1d441a59032295901a8a59bf1b10a0/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/75816ca1eac0bda33f5009aed8cef3df84f579ea17d30ee171e26c042935270b/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/e9baec640235591c576822b853cd0bece515fe6f0be3167695f776f55613f7f4/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/a9192ec67e68d14b8f82818c17ac5268cf92f30132cd92666a6fca0f798143cb/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/5377ebb6d33c92bd1ad9b7c2e397e06b2508dbdf7a8ee9435565c281c2d2e38c/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/712f21f1ca5d15b9872da625d17f0ab5079ce5bec3ba98cfe24f2c8c56794980/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/3f987fa8eecb7e93a56476d5f4cec980a1596a874424d734ac0d321039c30c9b/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/7cfd446b69a8c5bb17611ff789a872507a693f273dca79e9cce14a842d93591b/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/7fd61bd22ce1806cc00bd1ab6feb35a05a6afc4ae350edb823036755215a5471/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/f54c737b5f0b57eac5c1ee5c06144507b3db0cde2b242fdcbc14ca7895398bc9/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/61fc9bcd0bec95edb0db13d19435e06e6354d8ff348071bc305a649f657927f1/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/665161bc952ce767cef5e003b239855076e688816e8261ba3345d5fe1bb9705c/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/b744ef15642f688ea58a7d8e7b8d1f10cd67df1df6a0258f771993a40b15ef2c/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/72a214b49ae64a8c9f31c97a6839742ee2cc2e287bc447b830546a9f1146c48f/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/64b704cf7b0b588240c64191d3a4f223957fbff027ee430ed0ecdca91e0654fe/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/b58e24a04fce9e063a510f072589c08d63ff0f7ce6461098bbc33dc7102f915c/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/cee25066bc7c8445028c0124ecbce2dbb42491c9a496f2212185da6d62ab3c07/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/4f89f201af4d078564aad33d2d71b91027f3f75c7cc58607e05106cda1e0a926/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/9133087bb79a17249b38f9abde6831108c1d87cfd1021b8874e9bbc953a511ef/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/40d0b74c1c3834b4a20c971ec7edcc96aac31d409540640f69aa07f280e35da0/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/6041dffa8bc2fc14dc8bfc99f597a073bd2983b886a1807595327192ce3225fc/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/0e32becccf5d2bced6ed4ea0c3af52a55b8bddc01c7e94e7de8f8eefbbfbdae1/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/4aafc37c85ec1a006423d4f4a1dbfe7298efdd38e72567acb25cd10cc172cb2a/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/eb9a0b3a282aa88c91d851064cf50bf9f8a34a096e0f4626add202a3a98b25bc/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/2c0140ed68d78eee47d9babd317ffca440058f86d110d568b071d5096d9a1268/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/d60ea4bc2ce130de7dd9faea5141d6905f0b0314a155f4e96bb4f90aa1cc427e/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/485658ac56d39db4e2c17006a9f8d89fb1b2857900723917bc8aa3c0045be511/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/d8c01f6aced3e4b4feddf2629b4b144e7e9a7a0be1a29a15d3a77aff1eadd050/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/98afa15d4d92c34b7d93c1932df125a797e6330ad0feb6b53c93206a365abaf6/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/c08b9364b95daa653d48f442d392d216c7923da3a6c2e9ab67f347a97134d84d/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/39456ebe9a9bf90a0311b205566f6e0e70d54d70e5a18076c13a93fbbd51bc79/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/2e10714401387b6e07afaddd6a32cc186434efa590dfe0a5fe2139c9b711103f/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/adbd21e54cc0252ad1ebeeaf7b4af01b58dcdf90adddd58447645b53a8b130be/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/80a9042a6e0f6a9ca2fa7212af994bbcf6535c6b251c9b1edc8156dc3ab1c062/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/9390c99ad992565277a36f711702fbb4337ccc060b6c7f1367f372c3644f9548/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/4fa9415ee79c57a572a158ed52aa15e65ca5658da66593d504825a4abf49a39d/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/55729887f02614a9d08da04bbc1d95c900c844ed2c1888471f99d3e056297b43/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/b96c4e2f7b2354bc4f98deb9ada1e8ddac9486dcb4381b81f29379bc3487c28d/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/844d67243c3d44d9d614761f692033a0bc9a74eb462e363a0ef9c9c4c06e83b9/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/8b76c1cd50af8f4a14f2e08bca8eca6d20ad57563e5352b0b854042bfc7aea59/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/6f8124dedcdbe189b879d32d34d261474981d346eab78ccea33a083046e94c8a/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/555c5d0f3abacb3bc580ba07835dc0d507e3280fb90926c5833547c668e99b31/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/4b455b154092dfc6cbc85d2a7f3d77dbbc5293aff55b446ff4ad3d00dcb6066e/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.141964    6030 manager.go:217] Machine: {Timestamp:2026-10-30 19:32:01.141099254 +0000 UTC m=+0.056706103 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3118473 MemoryCapacity:32811999232 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aa4968c9b4781fc841f5e476389cb SystemUUID:ec2aa496-8c9b-4781-fc84-1f5e476389cb BootID:360cc010-2d66-47c7-b98e-fe1974cfc051 Filesystems:[{Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7a73dcc-6a78-4344-b0b3-7a8611466a41/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:354 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6b01178-0262-47a5-81bb-d715f5cf4380/volumes/kubernetes.io~projected/kube-api-access-bvm7j DeviceMajor:0 DeviceMinor:230 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d8aee8fb5c9641668f90f7de52260d610f8cf81bdeac276d590dfe26f80d536/userdata/shm DeviceMajor:0 DeviceMinor:277 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:284 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:289 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/be888c82-964b-42ae-948a-e6e27d276347/volumes/kubernetes.io~projected/kube-api-access-5xn78 DeviceMajor:0 DeviceMinor:355 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cdfe62e2-9594-4c48-a3bc-2dce9df78b56/volumes/kubernetes.io~projected/kube-api-access-4dgf7 DeviceMajor:0 DeviceMinor:60 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/a675f8c3-6d56-485d-8c87-438d66ebdc1e/volumes/kubernetes.io~projected/kube-api-access-xqggx DeviceMajor:0 DeviceMinor:204 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:282 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b5fcf354f1de5ae721c7e59bdbc57b009d0fcd8dffc411c85af9784f33c2318/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:290 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5aad5010-0827-4918-85d9-47e337a22774/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8eb8c11-ccd0-457f-92bd-c21db1c89e60/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:350 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fef01e5-9be4-4368-8f1e-3f73d7d6c60d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/a8eb8c11-ccd0-457f-92bd-c21db1c89e60/volumes/kubernetes.io~projected/kube-api-access-7c2bb DeviceMajor:0 DeviceMinor:59 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/d432cc60-02df-4a8d-94d4-93aafac213f2/volumes/kubernetes.io~projected/kube-api-access-lk9b4 DeviceMajor:0 DeviceMinor:67 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a1a321b62a9b9de7d0b59b69e9430c2a74a4f6b70b2d28b081282567e2300b4/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~projected/kube-api-access-7xn9h DeviceMajor:0 DeviceMinor:291 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe2bb906042c92e3f81991602e039cca05ede3fc631204df00ceba6e679577b5/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19eebbf5810a27b2a6adafe2b781d5d58f8a50f013fb9ac24422251ae6569f1e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/cedda70a-9baa-4592-818a-63092c854c8e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7f07af5464a3d7f7b8c28b7ca81681ec36e68037aa347554caf31757796be2c/userdata/shm DeviceMajor:0 DeviceMinor:300 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:290 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfda46fc-34fd-461d-a2bf-d4ec19596ed2/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/cfda46fc-34fd-461d-a2bf-d4ec19596ed2/volumes/kubernetes.io~projected/kube-api-access-8vpzr DeviceMajor:0 DeviceMinor:203 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aea6acd546a4bb51f2569595c8c3c6001c04c094b87b254989cbe76316a9c0eb/userdata/shm DeviceMajor:0 DeviceMinor:356 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ea2b374f029306389eb06c1bedfa13941608fc7e0fe310460a8d3b094709229/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92d09c10-51ac-4231-8195-d0c853887162/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:265 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:290 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee3c157f-683e-415d-b856-74da8b4fe669/volumes/kubernetes.io~projected/kube-api-access-kcfwl DeviceMajor:0 DeviceMinor:65 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/09163586b4ffabebea3176275992c748d561840fdc4417388c063f201ca008ac/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6b01178-0262-47a5-81bb-d715f5cf4380/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:229 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:288 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b16dc00a-c3b0-4767-a468-6d204ba44e55/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:283 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f59a3d2fa1f4bdde3a8fe04e2021f2bfc6b47b7e33ecd49e248c1665a784d27b/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16405999616 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cedda70a-9baa-4592-818a-63092c854c8e/volumes/kubernetes.io~projected/kube-api-access-4t74h DeviceMajor:0 DeviceMinor:202 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55ec1086b671b271d3dfa310dedfc2c6b40729d72cba4ca9141a1a58756155cd/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d2ce302416f14e67ff465ce723e3d0a21e9f1859dd8e1b26044efe09af53447/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/92d09c10-51ac-4231-8195-d0c853887162/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:269 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e97fc75dc8951d5cc89ff0ef02dd775bdcf7d5394b77e9c7a9a05bbd655a3b5/userdata/shm DeviceMajor:0 DeviceMinor:274 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e40c5103d22a472d26a9109f4f4897bb9cbb027fa6bbd4cc492602e19d223b2/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70833a700389847c63f703c4b20bba947f3be67b4805ea677d0a1b085ec704dd/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98ec24c765ccebd3f3ba457ae182a13e539380026cd528c65788a82e844adae3/userdata/shm DeviceMajor:0 DeviceMinor:360 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/deae8580d1249abde68c8681ce29ed814663d86a100c519e5bc3ca68198b88ed/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:290 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16405999616 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/b16dc00a-c3b0-4767-a468-6d204ba44e55/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4201a324-4bf9-46fc-b1be-c8381f3e9dba/volumes/kubernetes.io~projected/kube-api-access-g4jj8 DeviceMajor:0 DeviceMinor:272 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:288 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36320ef28ce33f5f0b43cc11e6ce6368151264e568b9b75c0e2b976772f7495d/userdata/shm DeviceMajor:0 DeviceMinor:358 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6ccdc6ef5808392e98132ecb9055e6479ae1eab7bf7ad98ee84375a6adbb6a5/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4201a324-4bf9-46fc-b1be-c8381f3e9dba/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:271 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/92d09c10-51ac-4231-8195-d0c853887162/volumes/kubernetes.io~projected/kube-api-access-cl4qt DeviceMajor:0 DeviceMinor:273 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:286 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5aad5010-0827-4918-85d9-47e337a22774/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99a447cd455cca759581ade9ee8a552300faff916254f55447b1a5ee5c6461a3/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35333d53-b864-432d-af6d-b1baec362af6/volumes/kubernetes.io~projected/kube-api-access-mbwlx DeviceMajor:0 DeviceMinor:61 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/b16dc00a-c3b0-4767-a468-6d204ba44e55/volumes/kubernetes.io~projected/kube-api-access-7bmxb DeviceMajor:0 DeviceMinor:63 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/5aad5010-0827-4918-85d9-47e337a22774/volumes/kubernetes.io~projected/kube-api-access-84w8n DeviceMajor:0 DeviceMinor:64 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a675f8c3-6d56-485d-8c87-438d66ebdc1e/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:285 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:287 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6035c60e-c58e-455e-a61f-19eb599272af/volumes/kubernetes.io~projected/kube-api-access-2q94g DeviceMajor:0 DeviceMinor:66 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef93f664-f7a4-42b9-bb9c-a10c519e9685/volumes/kubernetes.io~projected/kube-api-access-bb6r7 DeviceMajor:0 DeviceMinor:62 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6b01178-0262-47a5-81bb-d715f5cf4380/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/4201a324-4bf9-46fc-b1be-c8381f3e9dba/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:270 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:09163586b4ffabe MacAddress:2a:40:ab:86:70:6c Speed:10000 Mtu:8901} {Name:2e97fc75dc8951d MacAddress:7e:ec:e5:ea:0d:36 Speed:10000 Mtu:8901} {Name:36320ef28ce33f5 MacAddress:96:5f:93:08:f2:83 Speed:10000 Mtu:8901} {Name:4d2ce302416f14e MacAddress:96:2d:a1:e9:31:4e Speed:10000 Mtu:8901} {Name:70833a700389847 MacAddress:ae:a9:39:63:39:c6 Speed:10000 Mtu:8901} {Name:98ec24c765ccebd MacAddress:72:af:ef:0b:87:3e Speed:10000 Mtu:8901} {Name:99a447cd455cca7 MacAddress:ae:cf:9c:fb:4b:5d Speed:10000 Mtu:8901} {Name:aea6acd546a4bb5 MacAddress:aa:06:f7:be:9c:cd Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c1:a6:87:cd Speed:0 Mtu:9001} {Name:br-int MacAddress:42:fe:9d:ba:aa:61 Speed:0 Mtu:8901} {Name:d7f07af5464a3d7 MacAddress:6a:02:fd:4d:bf:dd Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c1:a6:87:cd Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:32:a9:18:43:3b:1d Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:de:9d:d5:69:ef:ea Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32811999232 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.142389    6030 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.142436    6030 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.142696    6030 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.142720    6030 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-167.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.142841    6030 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.142848    6030 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.142858    6030 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.142877    6030 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.142955    6030 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.143119    6030 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.143220    6030 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.143252    6030 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.143263    6030 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.143273    6030 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.143283    6030 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.143291    6030 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.143944    6030 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.143954    6030 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:32:01.146762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.144424    6030 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:32:01.147945 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.147904    6030 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:32:01.147945 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.147917    6030 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:32:01.148096 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.148081    6030 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 19:32:01.148305 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.148290    6030 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:32:01.148369 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.148330    6030 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:32:01.148457 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.148445    6030 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:32:01.148510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.148465    6030 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:32:01.148510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.148475    6030 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:32:01.148510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.148486    6030 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:32:01.148510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.148495    6030 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:32:01.148510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.148504    6030 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:32:01.148738 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.148514    6030 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:32:01.148738 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.148523    6030 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:32:01.148738 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.148535    6030 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:32:01.148738 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.148544    6030 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:32:01.148738 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.148557    6030 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:32:01.148738 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.148571    6030 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:32:01.148738 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.148610    6030 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:32:01.148738 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.148620    6030 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:32:01.152118 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.152104    6030 server.go:1263] "Started kubelet"
Oct 30 19:32:01.152118 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.152109    6030 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:32:01.152735 ip-10-0-142-167 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:32:01.152930 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.152860    6030 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:32:01.153025 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.152954    6030 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:32:01.153333 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.153316    6030 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:32:01.154404 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.154388    6030 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:32:01.158703 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.158685    6030 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:32:01.159156 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.159134    6030 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:32:01.159329 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.159312    6030 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:32:01.159384 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.159330    6030 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:32:01.159452 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:32:01.159434    6030 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-167.us-east-2.compute.internal\" not found"
Oct 30 19:32:01.159992 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.159949    6030 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:25:50 +0000 UTC" deadline="2028-04-23 07:59:46.05563244 +0000 UTC"
Oct 30 19:32:01.159992 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.159992    6030 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12972h27m44.895644448s"
Oct 30 19:32:01.160296 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.160039    6030 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:32:01.160739 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.160704    6030 factory.go:55] Registering systemd factory
Oct 30 19:32:01.160739 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.160731    6030 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:32:01.161464 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.161441    6030 factory.go:153] Registering CRI-O factory
Oct 30 19:32:01.161540 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.161468    6030 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:32:01.161540 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.161527    6030 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:32:01.161646 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.161546    6030 factory.go:103] Registering Raw factory
Oct 30 19:32:01.161646 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.161563    6030 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:32:01.161917 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.161901    6030 manager.go:319] Starting recovery of all containers
Oct 30 19:32:01.162936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.162888    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5aad5010-0827-4918-85d9-47e337a22774" volumeName="kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-env-overrides" seLinuxMountContext=""
Oct 30 19:32:01.163035 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.162940    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f098c7-fd9e-4ca8-85ef-412371228817" volumeName="kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:32:01.163035 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.162956    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f098c7-fd9e-4ca8-85ef-412371228817" volumeName="kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 19:32:01.163035 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.162972    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cedda70a-9baa-4592-818a-63092c854c8e" volumeName="kubernetes.io/secret/cedda70a-9baa-4592-818a-63092c854c8e-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 19:32:01.163035 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.162985    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6b01178-0262-47a5-81bb-d715f5cf4380" volumeName="kubernetes.io/secret/f6b01178-0262-47a5-81bb-d715f5cf4380-prometheus-operator-tls" seLinuxMountContext=""
Oct 30 19:32:01.163035 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.162997    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4201a324-4bf9-46fc-b1be-c8381f3e9dba" volumeName="kubernetes.io/configmap/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 19:32:01.163035 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163008    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4201a324-4bf9-46fc-b1be-c8381f3e9dba" volumeName="kubernetes.io/secret/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 19:32:01.163035 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163019    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4201a324-4bf9-46fc-b1be-c8381f3e9dba" volumeName="kubernetes.io/secret/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 19:32:01.163035 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163033    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfda46fc-34fd-461d-a2bf-d4ec19596ed2" volumeName="kubernetes.io/projected/cfda46fc-34fd-461d-a2bf-d4ec19596ed2-kube-api-access-8vpzr" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163062    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f098c7-fd9e-4ca8-85ef-412371228817" volumeName="kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-main-db" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163075    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92d09c10-51ac-4231-8195-d0c853887162" volumeName="kubernetes.io/empty-dir/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-textfile" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163087    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b16dc00a-c3b0-4767-a468-6d204ba44e55" volumeName="kubernetes.io/empty-dir/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-tuned" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163101    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cedda70a-9baa-4592-818a-63092c854c8e" volumeName="kubernetes.io/empty-dir/cedda70a-9baa-4592-818a-63092c854c8e-data-volume" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163112    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdfe62e2-9594-4c48-a3bc-2dce9df78b56" volumeName="kubernetes.io/projected/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-kube-api-access-4dgf7" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163127    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5aad5010-0827-4918-85d9-47e337a22774" volumeName="kubernetes.io/secret/5aad5010-0827-4918-85d9-47e337a22774-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163139    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f098c7-fd9e-4ca8-85ef-412371228817" volumeName="kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163150    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92d09c10-51ac-4231-8195-d0c853887162" volumeName="kubernetes.io/configmap/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163162    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8eb8c11-ccd0-457f-92bd-c21db1c89e60" volumeName="kubernetes.io/projected/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-kube-api-access-7c2bb" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163173    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4201a324-4bf9-46fc-b1be-c8381f3e9dba" volumeName="kubernetes.io/configmap/4201a324-4bf9-46fc-b1be-c8381f3e9dba-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163185    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92d09c10-51ac-4231-8195-d0c853887162" volumeName="kubernetes.io/projected/92d09c10-51ac-4231-8195-d0c853887162-kube-api-access-cl4qt" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163199    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8eb8c11-ccd0-457f-92bd-c21db1c89e60" volumeName="kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163211    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdfe62e2-9594-4c48-a3bc-2dce9df78b56" volumeName="kubernetes.io/configmap/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163221    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee3c157f-683e-415d-b856-74da8b4fe669" volumeName="kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163250    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f098c7-fd9e-4ca8-85ef-412371228817" volumeName="kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163288    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b16dc00a-c3b0-4767-a468-6d204ba44e55" volumeName="kubernetes.io/empty-dir/b16dc00a-c3b0-4767-a468-6d204ba44e55-tmp" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163301    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfda46fc-34fd-461d-a2bf-d4ec19596ed2" volumeName="kubernetes.io/secret/cfda46fc-34fd-461d-a2bf-d4ec19596ed2-cert" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163315    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d432cc60-02df-4a8d-94d4-93aafac213f2" volumeName="kubernetes.io/projected/d432cc60-02df-4a8d-94d4-93aafac213f2-kube-api-access-lk9b4" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163327    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7a73dcc-6a78-4344-b0b3-7a8611466a41" volumeName="kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163344    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cedda70a-9baa-4592-818a-63092c854c8e" volumeName="kubernetes.io/configmap/cedda70a-9baa-4592-818a-63092c854c8e-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163359    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee3c157f-683e-415d-b856-74da8b4fe669" volumeName="kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 19:32:01.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163370    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fef01e5-9be4-4368-8f1e-3f73d7d6c60d" volumeName="kubernetes.io/secret/0fef01e5-9be4-4368-8f1e-3f73d7d6c60d-agent-certs" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163381    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5aad5010-0827-4918-85d9-47e337a22774" volumeName="kubernetes.io/projected/5aad5010-0827-4918-85d9-47e337a22774-kube-api-access-84w8n" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163394    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92d09c10-51ac-4231-8195-d0c853887162" volumeName="kubernetes.io/secret/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163406    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a675f8c3-6d56-485d-8c87-438d66ebdc1e" volumeName="kubernetes.io/configmap/a675f8c3-6d56-485d-8c87-438d66ebdc1e-config-volume" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163417    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f098c7-fd9e-4ca8-85ef-412371228817" volumeName="kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-config-out" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163428    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f098c7-fd9e-4ca8-85ef-412371228817" volumeName="kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163441    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f098c7-fd9e-4ca8-85ef-412371228817" volumeName="kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-config-volume" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163451    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a675f8c3-6d56-485d-8c87-438d66ebdc1e" volumeName="kubernetes.io/projected/a675f8c3-6d56-485d-8c87-438d66ebdc1e-kube-api-access-xqggx" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163465    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a675f8c3-6d56-485d-8c87-438d66ebdc1e" volumeName="kubernetes.io/secret/a675f8c3-6d56-485d-8c87-438d66ebdc1e-metrics-tls" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163478    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b16dc00a-c3b0-4767-a468-6d204ba44e55" volumeName="kubernetes.io/projected/b16dc00a-c3b0-4767-a468-6d204ba44e55-kube-api-access-7bmxb" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163495    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef93f664-f7a4-42b9-bb9c-a10c519e9685" volumeName="kubernetes.io/projected/ef93f664-f7a4-42b9-bb9c-a10c519e9685-kube-api-access-bb6r7" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163508    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f098c7-fd9e-4ca8-85ef-412371228817" volumeName="kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163520    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4201a324-4bf9-46fc-b1be-c8381f3e9dba" volumeName="kubernetes.io/empty-dir/4201a324-4bf9-46fc-b1be-c8381f3e9dba-volume-directive-shadow" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163530    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4201a324-4bf9-46fc-b1be-c8381f3e9dba" volumeName="kubernetes.io/projected/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-api-access-g4jj8" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163542    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdfe62e2-9594-4c48-a3bc-2dce9df78b56" volumeName="kubernetes.io/configmap/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-daemon-config" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163553    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cedda70a-9baa-4592-818a-63092c854c8e" volumeName="kubernetes.io/projected/cedda70a-9baa-4592-818a-63092c854c8e-kube-api-access-4t74h" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163564    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35333d53-b864-432d-af6d-b1baec362af6" volumeName="kubernetes.io/projected/35333d53-b864-432d-af6d-b1baec362af6-kube-api-access-mbwlx" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163576    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6035c60e-c58e-455e-a61f-19eb599272af" volumeName="kubernetes.io/projected/6035c60e-c58e-455e-a61f-19eb599272af-kube-api-access-2q94g" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163589    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f098c7-fd9e-4ca8-85ef-412371228817" volumeName="kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-cluster-tls-config" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163600    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a675f8c3-6d56-485d-8c87-438d66ebdc1e" volumeName="kubernetes.io/empty-dir/a675f8c3-6d56-485d-8c87-438d66ebdc1e-tmp-dir" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163613    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f098c7-fd9e-4ca8-85ef-412371228817" volumeName="kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-web-config" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163625    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92d09c10-51ac-4231-8195-d0c853887162" volumeName="kubernetes.io/configmap/92d09c10-51ac-4231-8195-d0c853887162-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163640    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92d09c10-51ac-4231-8195-d0c853887162" volumeName="kubernetes.io/secret/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-tls" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163654    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be888c82-964b-42ae-948a-e6e27d276347" volumeName="kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163669    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d432cc60-02df-4a8d-94d4-93aafac213f2" volumeName="kubernetes.io/configmap/d432cc60-02df-4a8d-94d4-93aafac213f2-iptables-alerter-script" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163681    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef93f664-f7a4-42b9-bb9c-a10c519e9685" volumeName="kubernetes.io/empty-dir/ef93f664-f7a4-42b9-bb9c-a10c519e9685-tmp-dir" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163705    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6b01178-0262-47a5-81bb-d715f5cf4380" volumeName="kubernetes.io/projected/f6b01178-0262-47a5-81bb-d715f5cf4380-kube-api-access-bvm7j" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163720    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fef01e5-9be4-4368-8f1e-3f73d7d6c60d" volumeName="kubernetes.io/configmap/0fef01e5-9be4-4368-8f1e-3f73d7d6c60d-konnectivity-ca" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163733    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f098c7-fd9e-4ca8-85ef-412371228817" volumeName="kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-tls-assets" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163748    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee3c157f-683e-415d-b856-74da8b4fe669" volumeName="kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163761    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35333d53-b864-432d-af6d-b1baec362af6" volumeName="kubernetes.io/configmap/35333d53-b864-432d-af6d-b1baec362af6-serviceca" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163779    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5aad5010-0827-4918-85d9-47e337a22774" volumeName="kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-ovnkube-config" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163791    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5aad5010-0827-4918-85d9-47e337a22774" volumeName="kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163804    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f098c7-fd9e-4ca8-85ef-412371228817" volumeName="kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-kube-api-access-7xn9h" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163816    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee3c157f-683e-415d-b856-74da8b4fe669" volumeName="kubernetes.io/projected/ee3c157f-683e-415d-b856-74da8b4fe669-kube-api-access-kcfwl" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163828    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6b01178-0262-47a5-81bb-d715f5cf4380" volumeName="kubernetes.io/configmap/f6b01178-0262-47a5-81bb-d715f5cf4380-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163840    6030 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6b01178-0262-47a5-81bb-d715f5cf4380" volumeName="kubernetes.io/secret/f6b01178-0262-47a5-81bb-d715f5cf4380-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163852    6030 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:32:01.164321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.163860    6030 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:32:01.173571 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.169763    6030 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:32:01.175305 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.175286    6030 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 19:32:01.175893 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.175871    6030 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:32:01.191243 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.191190    6030 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:32:01.192937 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.192911    6030 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:32:01.192937 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.192936    6030 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:32:01.193139 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.192956    6030 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:32:01.193139 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:32:01.192995    6030 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 19:32:01.198235 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.198201    6030 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:32:01.203071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.203024    6030 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:32:01.206620 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.206585    6030 generic.go:356] "Generic (PLEG): container finished" podID="87f098c7-fd9e-4ca8-85ef-412371228817" containerID="3de25613b3637d81768b30d3e67b2c05d438c51ccf34e834f06ad459f38dbd6c" exitCode=0
Oct 30 19:32:01.218297 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.218263    6030 generic.go:356] "Generic (PLEG): container finished" podID="ee3c157f-683e-415d-b856-74da8b4fe669" containerID="6a9bd36648ae2133d8b9d1cf320819f238f297e0950ddd16c5ed756c79a5e94d" exitCode=0
Oct 30 19:32:01.218297 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.218292    6030 generic.go:356] "Generic (PLEG): container finished" podID="ee3c157f-683e-415d-b856-74da8b4fe669" containerID="49cbc617eec9bfa372e6a9840a85c7f1ee504f2971f1d6dbbeeb406f1e40c1c4" exitCode=0
Oct 30 19:32:01.218297 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.218301    6030 generic.go:356] "Generic (PLEG): container finished" podID="ee3c157f-683e-415d-b856-74da8b4fe669" containerID="d2317b6b9331f8979be1a78659d9111859a8e1ce85fdbdd59c0276d8404924e8" exitCode=0
Oct 30 19:32:01.218297 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.218310    6030 generic.go:356] "Generic (PLEG): container finished" podID="ee3c157f-683e-415d-b856-74da8b4fe669" containerID="02d24d5b5a11a7daff8c46ae0c2e5145de8b599afb53a33e821abb1be5fd7fbc" exitCode=0
Oct 30 19:32:01.218630 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.218319    6030 generic.go:356] "Generic (PLEG): container finished" podID="ee3c157f-683e-415d-b856-74da8b4fe669" containerID="291ffdce5b0e571e858c4783ae9e8295b6fd0919af05e254934a075cb06e962b" exitCode=0
Oct 30 19:32:01.218630 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.218327    6030 generic.go:356] "Generic (PLEG): container finished" podID="ee3c157f-683e-415d-b856-74da8b4fe669" containerID="4cede867c67d294b4cb8f31476c8b3bd5c1827a5b15c9dce8e8bdb17fd0b47a6" exitCode=0
Oct 30 19:32:01.231487 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.231457    6030 generic.go:356] "Generic (PLEG): container finished" podID="b1c62c9f273bf0b4293082033ce5e525" containerID="ac0d3a23121a795127e9043d29dd14f983ba4cf8c27d503d082b18a2930d0014" exitCode=0
Oct 30 19:32:01.233656 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.233624    6030 generic.go:356] "Generic (PLEG): container finished" podID="92d09c10-51ac-4231-8195-d0c853887162" containerID="897dbe6ce0ea8ee1149a909c91506e74468871ebec76e706cc39087d87f3612a" exitCode=0
Oct 30 19:32:01.257010 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.256985    6030 manager.go:324] Recovery completed
Oct 30 19:32:01.276936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.276871    6030 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:32:01.276936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.276886    6030 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:32:01.276936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.276905    6030 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:32:01.277150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.277025    6030 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 19:32:01.277150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.277035    6030 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 19:32:01.277150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.277076    6030 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 19:32:01.277150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.277083    6030 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 19:32:01.277150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.277089    6030 policy_none.go:50] "Start"
Oct 30 19:32:01.277150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.277097    6030 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:32:01.277150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.277107    6030 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:32:01.277347 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.277248    6030 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 19:32:01.277347 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.277256    6030 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 19:32:01.277347 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.277262    6030 policy_none.go:44] "Start"
Oct 30 19:32:01.282628 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.282578    6030 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:32:01.286291 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.286274    6030 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:32:01.286742 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:32:01.286301    6030 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:32:01.286742 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.286743    6030 server.go:87] "Starting device plugin registration server"
Oct 30 19:32:01.287014 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.287004    6030 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:32:01.287066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.287016    6030 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:32:01.287129 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.287102    6030 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:32:01.287257 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.287210    6030 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:32:01.287257 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.287253    6030 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:32:01.287257 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.287258    6030 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:32:01.287380 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.287304    6030 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:32:01.287243548Z","UUID":"f835d800-04de-417c-80f8-7d37e5831255","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:32:01.288881 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:32:01.288861    6030 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:32:01.289083 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.288900    6030 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:32:01.289195 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.289102    6030 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:32:01.293108 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.293067    6030 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal"]
Oct 30 19:32:01.293708 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.293665    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal" event={"ID":"2bdf808689747c42465325897cee67ca","Type":"ContainerStarted","Data":"b75f9978b8fef6a0601c6fd405797cd1b2260fec1308e353dd04acc0c6060162"}
Oct 30 19:32:01.293708 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.293710    6030 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:01.293708 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.293716    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal" event={"ID":"2bdf808689747c42465325897cee67ca","Type":"ContainerStarted","Data":"19eebbf5810a27b2a6adafe2b781d5d58f8a50f013fb9ac24422251ae6569f1e"}
Oct 30 19:32:01.293929 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.293830    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal" event={"ID":"b1c62c9f273bf0b4293082033ce5e525","Type":"ContainerStarted","Data":"487d2240e2fa53ecdc8599bbb1e0d686e0400c70a9c5c78c041056893b0af133"}
Oct 30 19:32:01.293929 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.293853    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal" event={"ID":"b1c62c9f273bf0b4293082033ce5e525","Type":"ContainerDied","Data":"ac0d3a23121a795127e9043d29dd14f983ba4cf8c27d503d082b18a2930d0014"}
Oct 30 19:32:01.293929 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.293873    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal" event={"ID":"b1c62c9f273bf0b4293082033ce5e525","Type":"ContainerStarted","Data":"e6ccdc6ef5808392e98132ecb9055e6479ae1eab7bf7ad98ee84375a6adbb6a5"}
Oct 30 19:32:01.293929 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.293889    6030 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:01.328777 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.328746    6030 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:32:01.328932 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:32:01.328801    6030 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:01.328932 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.328809    6030 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:32:01.328932 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:32:01.328847    6030 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:01.375897 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.375856    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1c62c9f273bf0b4293082033ce5e525-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"b1c62c9f273bf0b4293082033ce5e525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:01.375897 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.375900    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1c62c9f273bf0b4293082033ce5e525-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"b1c62c9f273bf0b4293082033ce5e525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:01.376124 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.375924    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2bdf808689747c42465325897cee67ca-config\") pod \"kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"2bdf808689747c42465325897cee67ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:01.388967 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.388942    6030 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:32:01.391866 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.391848    6030 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:32:01.391975 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.391877    6030 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:32:01.391975 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.391891    6030 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-167.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:32:01.391975 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.391933    6030 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:01.410688 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.410667    6030 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:01.410758 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.410724    6030 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:01.476098 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.476069    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1c62c9f273bf0b4293082033ce5e525-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"b1c62c9f273bf0b4293082033ce5e525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:01.476259 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.476105    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1c62c9f273bf0b4293082033ce5e525-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"b1c62c9f273bf0b4293082033ce5e525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:01.476259 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.476124    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2bdf808689747c42465325897cee67ca-config\") pod \"kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"2bdf808689747c42465325897cee67ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:01.476259 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.476185    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1c62c9f273bf0b4293082033ce5e525-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"b1c62c9f273bf0b4293082033ce5e525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:01.476259 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.476234    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1c62c9f273bf0b4293082033ce5e525-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"b1c62c9f273bf0b4293082033ce5e525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:01.476390 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:01.476265    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2bdf808689747c42465325897cee67ca-config\") pod \"kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal\" (UID: \"2bdf808689747c42465325897cee67ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:02.143649 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.143600    6030 apiserver.go:52] "Watching apiserver"
Oct 30 19:32:02.156160 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.156134    6030 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:32:02.156678 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.156654    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp","kube-system/global-pull-secret-syncer-6vm8t","kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-4pwl6","openshift-multus/network-metrics-daemon-xmzrn","kube-system/konnectivity-agent-qnmn2","openshift-monitoring/alertmanager-main-0","openshift-dns/node-resolver-dwfmh","openshift-multus/multus-additional-cni-plugins-64whk","openshift-network-diagnostics/network-check-target-hr4sv","openshift-dns/dns-default-pzxdj","openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5","openshift-image-registry/node-ca-bxxz2","openshift-network-operator/iptables-alerter-45rm5","openshift-monitoring/node-exporter-4jpzb","openshift-ovn-kubernetes/ovnkube-node-7rv7g","openshift-multus/multus-vjc5d","openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp","openshift-ingress-canary/ingress-canary-crfs6","openshift-insights/insights-runtime-extractor-xbxhg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"]
Oct 30 19:32:02.163137 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.163100    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vcqml\""
Oct 30 19:32:02.163299 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.163227    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:32:02.163803 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.163745    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jznq8\""
Oct 30 19:32:02.163944 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.163863    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g4c9f\""
Oct 30 19:32:02.164356 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.164309    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:32:02.164516 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.164502    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-x9qfs\""
Oct 30 19:32:02.164611 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.164587    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:32:02.164928 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.164911    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:32:02.165065 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.165027    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:32:02.165138 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.165091    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:32:02.165184 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.165175    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:32:02.165250 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.165222    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:32:02.165297 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.165281    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:32:02.165425 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.165408    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:32:02.165509 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.165028    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:32:02.165609 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.165550    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:32:02.165609 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.165579    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:32:02.165767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.165634    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7c6bv\""
Oct 30 19:32:02.165767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.165667    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:32:02.166138 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166085    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:32:02.166440 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166413    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:32:02.166543 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166509    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:32:02.166597 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166550    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:32:02.166758 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166740    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:32:02.166870 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166794    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:32:02.166870 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166809    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:32:02.166870 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166845    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rpkw7\""
Oct 30 19:32:02.166870 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166158    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5r9wt\""
Oct 30 19:32:02.167086 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166744    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4mthb\""
Oct 30 19:32:02.167086 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166900    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:32:02.167086 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166765    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:32:02.167086 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166914    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:32:02.167086 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166922    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:32:02.167086 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166181    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:32:02.167086 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166101    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wr97x\""
Oct 30 19:32:02.167086 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166135    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:32:02.167086 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.166992    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:32:02.167528 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.167150    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:32:02.167618 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.167583    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:32:02.167618 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.167604    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-79gww\""
Oct 30 19:32:02.167755 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.167644    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:32:02.167755 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.167650    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h46gl\""
Oct 30 19:32:02.167755 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.167584    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:32:02.167755 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.167729    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:32:02.167755 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.167742    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ztltl\""
Oct 30 19:32:02.167985 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.167772    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:32:02.167985 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.167839    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7mr7r\""
Oct 30 19:32:02.167985 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.167851    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-scxl6\""
Oct 30 19:32:02.167985 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.167854    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:32:02.167985 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.167903    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:32:02.168944 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.168923    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:32:02.171426 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.171409    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:32:02.171780 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.171764    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:32:02.171780 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.171767    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f2mgm\""
Oct 30 19:32:02.171905 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.171783    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:32:02.171905 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.171850    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 19:32:02.172103 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.172089    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 19:32:02.172103 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.172103    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:32:02.172206 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.172184    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-q5q65\""
Oct 30 19:32:02.172764 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.172745    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 19:32:02.172820 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.172793    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 19:32:02.173149 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.173135    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:32:02.173194 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.173187    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6xbb8\""
Oct 30 19:32:02.173253 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.173242    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vzztc\""
Oct 30 19:32:02.173295 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.173257    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:32:02.173295 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.173266    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:32:02.173360 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.173306    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 19:32:02.173360 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.173353    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:32:02.173894 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.173879    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 19:32:02.173937 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.173909    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 19:32:02.173937 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.173913    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:32:02.174204 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.174192    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hwkcq\""
Oct 30 19:32:02.174270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.174255    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 19:32:02.174332 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.174300    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 19:32:02.174332 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.174303    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 19:32:02.174396 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.174360    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 19:32:02.175372 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.175359    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 19:32:02.176099 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.176088    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 19:32:02.177776 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.177762    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 19:32:02.178540 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.178525    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fef01e5-9be4-4368-8f1e-3f73d7d6c60d-konnectivity-ca\") pod \"konnectivity-agent-qnmn2\" (UID: \"0fef01e5-9be4-4368-8f1e-3f73d7d6c60d\") " pod="kube-system/konnectivity-agent-qnmn2"
Oct 30 19:32:02.178575 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.178551    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-socket-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.178575 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.178566    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-cni-binary-copy\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:32:02.178648 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.178581    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-multus-certs\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.178648 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.178597    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dgf7\" (UniqueName: \"kubernetes.io/projected/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-kube-api-access-4dgf7\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.178715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.178700    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-kubelet\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.178796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.178777    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.178826 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.178794    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fef01e5-9be4-4368-8f1e-3f73d7d6c60d-konnectivity-ca\") pod \"konnectivity-agent-qnmn2\" (UID: \"0fef01e5-9be4-4368-8f1e-3f73d7d6c60d\") " pod="kube-system/konnectivity-agent-qnmn2"
Oct 30 19:32:02.178861 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.178831    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-registration-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.178910 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.178873    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-cni-binary-copy\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:32:02.178910 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.178877    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-sys-fs\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.178975 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.178917    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-daemon-config\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.178975 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.178938    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-run-ovn-kubernetes\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.178975 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.178969    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs\") pod \"network-metrics-daemon-xmzrn\" (UID: \"a8eb8c11-ccd0-457f-92bd-c21db1c89e60\") " pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:32:02.179104 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.178991    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2bb\" (UniqueName: \"kubernetes.io/projected/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-kube-api-access-7c2bb\") pod \"network-metrics-daemon-xmzrn\" (UID: \"a8eb8c11-ccd0-457f-92bd-c21db1c89e60\") " pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:32:02.179104 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179081    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-cni-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.179173 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179119    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-daemon-config\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.179222 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179178    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-socket-dir-parent\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.179222 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179210    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-cni-multus\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.179291 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179255    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-modprobe-d\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.179291 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179265    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-metrics-certs\") pod \"network-metrics-daemon-xmzrn\" (UID: \"a8eb8c11-ccd0-457f-92bd-c21db1c89e60\") " pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:32:02.179291 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179279    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-ovnkube-script-lib\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.179379 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179313    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-cnibin\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:32:02.179379 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179329    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-cnibin\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.179379 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179366    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b16dc00a-c3b0-4767-a468-6d204ba44e55-tmp\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.179515 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179386    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef93f664-f7a4-42b9-bb9c-a10c519e9685-hosts-file\") pod \"node-resolver-dwfmh\" (UID: \"ef93f664-f7a4-42b9-bb9c-a10c519e9685\") " pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:32:02.179515 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179401    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-ovnkube-config\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.179515 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179445    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-ovnkube-script-lib\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.179515 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179469    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fef01e5-9be4-4368-8f1e-3f73d7d6c60d-agent-certs\") pod \"konnectivity-agent-qnmn2\" (UID: \"0fef01e5-9be4-4368-8f1e-3f73d7d6c60d\") " pod="kube-system/konnectivity-agent-qnmn2"
Oct 30 19:32:02.179515 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179465    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b16dc00a-c3b0-4767-a468-6d204ba44e55-tmp\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.179515 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179486    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-etc-selinux\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.179796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179525    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-hostroot\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.179796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179552    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-run\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.179796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179571    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-ovnkube-config\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.179796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179574    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-lib-modules\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.179796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179598    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-run-netns\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.179796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179627    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5aad5010-0827-4918-85d9-47e337a22774-ovn-node-metrics-cert\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.179796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179658    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-tuning-conf-dir\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:32:02.179796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179677    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysctl-d\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.179796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179696    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d432cc60-02df-4a8d-94d4-93aafac213f2-host-slash\") pod \"iptables-alerter-45rm5\" (UID: \"d432cc60-02df-4a8d-94d4-93aafac213f2\") " pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:32:02.179796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179712    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-slash\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.179796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179747    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fef01e5-9be4-4368-8f1e-3f73d7d6c60d-agent-certs\") pod \"konnectivity-agent-qnmn2\" (UID: \"0fef01e5-9be4-4368-8f1e-3f73d7d6c60d\") " pod="kube-system/konnectivity-agent-qnmn2"
Oct 30 19:32:02.179796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179784    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179807    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-system-cni-dir\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179823    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179847    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5aad5010-0827-4918-85d9-47e337a22774-ovn-node-metrics-cert\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179849    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179877    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcfwl\" (UniqueName: \"kubernetes.io/projected/ee3c157f-683e-415d-b856-74da8b4fe669-kube-api-access-kcfwl\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179895    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-netns\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.179915    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-tuned\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180022    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-ovn\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180032    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180058    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-cni-bin\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180065    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee3c157f-683e-415d-b856-74da8b4fe669-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180083    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysconfig\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180099    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-tuned\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180106    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-host\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180134    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180156    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb6r7\" (UniqueName: \"kubernetes.io/projected/ef93f664-f7a4-42b9-bb9c-a10c519e9685-kube-api-access-bb6r7\") pod \"node-resolver-dwfmh\" (UID: \"ef93f664-f7a4-42b9-bb9c-a10c519e9685\") " pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180181    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xn78\" (UniqueName: \"kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78\") pod \"network-check-target-hr4sv\" (UID: \"be888c82-964b-42ae-948a-e6e27d276347\") " pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180197    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-system-cni-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180243    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-cni-binary-copy\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.180270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180268    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk9b4\" (UniqueName: \"kubernetes.io/projected/d432cc60-02df-4a8d-94d4-93aafac213f2-kube-api-access-lk9b4\") pod \"iptables-alerter-45rm5\" (UID: \"d432cc60-02df-4a8d-94d4-93aafac213f2\") " pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:32:02.180855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180410    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-systemd\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.180855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180432    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84w8n\" (UniqueName: \"kubernetes.io/projected/5aad5010-0827-4918-85d9-47e337a22774-kube-api-access-84w8n\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.180855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180489    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-kubelet\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.180855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180551    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-env-overrides\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.180855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180555    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-cni-binary-copy\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.180855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180628    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-sys\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.180855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180652    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bmxb\" (UniqueName: \"kubernetes.io/projected/b16dc00a-c3b0-4767-a468-6d204ba44e55-kube-api-access-7bmxb\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.180855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180671    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-var-lib-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.180855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180693    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5aad5010-0827-4918-85d9-47e337a22774-env-overrides\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.180855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180706    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-cni-netd\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.180855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180729    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q94g\" (UniqueName: \"kubernetes.io/projected/6035c60e-c58e-455e-a61f-19eb599272af-kube-api-access-2q94g\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.180855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180747    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-cni-bin\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.180855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180820    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-etc-kubernetes\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.180855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180852    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-etc-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.181398 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180881    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef93f664-f7a4-42b9-bb9c-a10c519e9685-tmp-dir\") pod \"node-resolver-dwfmh\" (UID: \"ef93f664-f7a4-42b9-bb9c-a10c519e9685\") " pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:32:02.181398 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180907    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-k8s-cni-cncf-io\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.181398 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180939    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysctl-conf\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.181398 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180963    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-var-lib-kubelet\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.181398 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.180966    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef93f664-f7a4-42b9-bb9c-a10c519e9685-tmp-dir\") pod \"node-resolver-dwfmh\" (UID: \"ef93f664-f7a4-42b9-bb9c-a10c519e9685\") " pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:32:02.181398 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.181014    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-systemd-units\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.181398 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.181077    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-device-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.181398 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.181102    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-conf-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.181398 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.181127    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-kubernetes\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.181398 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.181149    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-systemd\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.181398 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.181178    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d432cc60-02df-4a8d-94d4-93aafac213f2-iptables-alerter-script\") pod \"iptables-alerter-45rm5\" (UID: \"d432cc60-02df-4a8d-94d4-93aafac213f2\") " pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:32:02.181398 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.181201    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-node-log\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.181398 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.181226    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-log-socket\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.181398 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.181330    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d432cc60-02df-4a8d-94d4-93aafac213f2-iptables-alerter-script\") pod \"iptables-alerter-45rm5\" (UID: \"d432cc60-02df-4a8d-94d4-93aafac213f2\") " pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:32:02.190693 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.190672    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcfwl\" (UniqueName: \"kubernetes.io/projected/ee3c157f-683e-415d-b856-74da8b4fe669-kube-api-access-kcfwl\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:32:02.190739 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.190697    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dgf7\" (UniqueName: \"kubernetes.io/projected/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-kube-api-access-4dgf7\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.192128 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.192090    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb6r7\" (UniqueName: \"kubernetes.io/projected/ef93f664-f7a4-42b9-bb9c-a10c519e9685-kube-api-access-bb6r7\") pod \"node-resolver-dwfmh\" (UID: \"ef93f664-f7a4-42b9-bb9c-a10c519e9685\") " pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:32:02.192246 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.192187    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c2bb\" (UniqueName: \"kubernetes.io/projected/a8eb8c11-ccd0-457f-92bd-c21db1c89e60-kube-api-access-7c2bb\") pod \"network-metrics-daemon-xmzrn\" (UID: \"a8eb8c11-ccd0-457f-92bd-c21db1c89e60\") " pod="openshift-multus/network-metrics-daemon-xmzrn"
Oct 30 19:32:02.192246 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.192193    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xn78\" (UniqueName: \"kubernetes.io/projected/be888c82-964b-42ae-948a-e6e27d276347-kube-api-access-5xn78\") pod \"network-check-target-hr4sv\" (UID: \"be888c82-964b-42ae-948a-e6e27d276347\") " pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:32:02.192246 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.192202    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84w8n\" (UniqueName: \"kubernetes.io/projected/5aad5010-0827-4918-85d9-47e337a22774-kube-api-access-84w8n\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.192246 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.192188    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk9b4\" (UniqueName: \"kubernetes.io/projected/d432cc60-02df-4a8d-94d4-93aafac213f2-kube-api-access-lk9b4\") pod \"iptables-alerter-45rm5\" (UID: \"d432cc60-02df-4a8d-94d4-93aafac213f2\") " pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:32:02.192246 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.192239    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bmxb\" (UniqueName: \"kubernetes.io/projected/b16dc00a-c3b0-4767-a468-6d204ba44e55-kube-api-access-7bmxb\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.192394 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.192239    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q94g\" (UniqueName: \"kubernetes.io/projected/6035c60e-c58e-455e-a61f-19eb599272af-kube-api-access-2q94g\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.250384 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.250308    6030 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:02.250613 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.250481    6030 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:02.261285 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.261262    6030 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:32:02.264112 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.264088    6030 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:32:02.264222 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:32:02.264141    6030 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:02.264222 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.264202    6030 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:32:02.264352 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:32:02.264246    6030 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.us-east-2.compute.internal"
Oct 30 19:32:02.281683 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.281648    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-registration-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.281809 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.281686    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-sys-fs\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.281809 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.281711    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a675f8c3-6d56-485d-8c87-438d66ebdc1e-config-volume\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:32:02.281809 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.281735    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.281809 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.281758    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-kubelet\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.281809 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.281779    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.281809 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.281801    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfda46fc-34fd-461d-a2bf-d4ec19596ed2-cert\") pod \"ingress-canary-crfs6\" (UID: \"cfda46fc-34fd-461d-a2bf-d4ec19596ed2\") " pod="openshift-ingress-canary/ingress-canary-crfs6"
Oct 30 19:32:02.282133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.281826    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4201a324-4bf9-46fc-b1be-c8381f3e9dba-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:32:02.282133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.281852    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-run-ovn-kubernetes\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.282133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.281870    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-kubelet\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.282133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.281880    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6b01178-0262-47a5-81bb-d715f5cf4380-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:32:02.282133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.281929    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-socket-dir-parent\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.282133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.281947    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-cni-multus\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.282133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.281966    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a675f8c3-6d56-485d-8c87-438d66ebdc1e-config-volume\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:32:02.282133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.281993    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.282133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282013    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-sys-fs\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.282133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.281966    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:32:02.282133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.281795    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-registration-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.282133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282065    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.282133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282076    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.282133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282108    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.282133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282137    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6b01178-0262-47a5-81bb-d715f5cf4380-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282139    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cedda70a-9baa-4592-818a-63092c854c8e-crio-socket\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282185    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92d09c10-51ac-4231-8195-d0c853887162-metrics-client-ca\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282212    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-hostroot\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282217    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-cni-multus\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282225    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282235    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-run\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282239    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4201a324-4bf9-46fc-b1be-c8381f3e9dba-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282260    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d432cc60-02df-4a8d-94d4-93aafac213f2-host-slash\") pod \"iptables-alerter-45rm5\" (UID: \"d432cc60-02df-4a8d-94d4-93aafac213f2\") " pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282253    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfda46fc-34fd-461d-a2bf-d4ec19596ed2-cert\") pod \"ingress-canary-crfs6\" (UID: \"cfda46fc-34fd-461d-a2bf-d4ec19596ed2\") " pod="openshift-ingress-canary/ingress-canary-crfs6"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282258    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-run-ovn-kubernetes\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282296    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-socket-dir-parent\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282305    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-slash\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282332    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282386    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282412    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-hostroot\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282456    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-slash\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282503    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d432cc60-02df-4a8d-94d4-93aafac213f2-host-slash\") pod \"iptables-alerter-45rm5\" (UID: \"d432cc60-02df-4a8d-94d4-93aafac213f2\") " pod="openshift-network-operator/iptables-alerter-45rm5"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282540    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-run\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282560    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7a73dcc-6a78-4344-b0b3-7a8611466a41-dbus\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282580    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vpzr\" (UniqueName: \"kubernetes.io/projected/cfda46fc-34fd-461d-a2bf-d4ec19596ed2-kube-api-access-8vpzr\") pod \"ingress-canary-crfs6\" (UID: \"cfda46fc-34fd-461d-a2bf-d4ec19596ed2\") " pod="openshift-ingress-canary/ingress-canary-crfs6"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282583    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282592    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92d09c10-51ac-4231-8195-d0c853887162-metrics-client-ca\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282604    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-sys\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282664    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-ovn\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282758    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-cni-bin\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.282794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282788    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cedda70a-9baa-4592-818a-63092c854c8e-data-volume\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282818    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-ovn\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282827    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282846    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-cni-bin\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282859    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-wtmp\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282909    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cedda70a-9baa-4592-818a-63092c854c8e-data-volume\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282953    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cedda70a-9baa-4592-818a-63092c854c8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.282988    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-system-cni-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283072    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-system-cni-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283090    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-systemd\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283123    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvm7j\" (UniqueName: \"kubernetes.io/projected/f6b01178-0262-47a5-81bb-d715f5cf4380-kube-api-access-bvm7j\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283149    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-kubelet\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283171    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cedda70a-9baa-4592-818a-63092c854c8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283174    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-tls-assets\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283125    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-systemd\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283200    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-sys\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283224    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-cni-netd\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283236    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-kubelet\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283252    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-cni-bin\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283265    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-cni-netd\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283272    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-sys\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283279    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-etc-kubernetes\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283291    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-var-lib-cni-bin\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283307    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7a73dcc-6a78-4344-b0b3-7a8611466a41-kubelet-config\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283329    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-device-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283329    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-tls-assets\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283330    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-etc-kubernetes\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283372    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283416    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-config-out\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283444    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-device-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283500    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283580    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-config-out\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283604    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283609    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283634    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283659    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-kubernetes\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283665    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283673    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283685    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-systemd\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283719    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-node-log\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283725    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-kubernetes\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283738    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-systemd\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283743    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-log-socket\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283769    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-socket-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283777    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-node-log\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283821    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-log-socket\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283829    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283836    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-socket-dir\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283850    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-multus-certs\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283876    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283895    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283918    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283878    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-multus-certs\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.283920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283942    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35333d53-b864-432d-af6d-b1baec362af6-host\") pod \"node-ca-bxxz2\" (UID: \"35333d53-b864-432d-af6d-b1baec362af6\") " pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283967    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwlx\" (UniqueName: \"kubernetes.io/projected/35333d53-b864-432d-af6d-b1baec362af6-kube-api-access-mbwlx\") pod \"node-ca-bxxz2\" (UID: \"35333d53-b864-432d-af6d-b1baec362af6\") " pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.283992    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-cni-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284015    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-modprobe-d\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284079    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqggx\" (UniqueName: \"kubernetes.io/projected/a675f8c3-6d56-485d-8c87-438d66ebdc1e-kube-api-access-xqggx\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284099    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284108    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-textfile\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284121    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-cni-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284139    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-cnibin\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284144    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-modprobe-d\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284165    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-cnibin\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284186    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-cnibin\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284189    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef93f664-f7a4-42b9-bb9c-a10c519e9685-hosts-file\") pod \"node-resolver-dwfmh\" (UID: \"ef93f664-f7a4-42b9-bb9c-a10c519e9685\") " pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284197    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-textfile\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284184    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284227    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef93f664-f7a4-42b9-bb9c-a10c519e9685-hosts-file\") pod \"node-resolver-dwfmh\" (UID: \"ef93f664-f7a4-42b9-bb9c-a10c519e9685\") " pod="openshift-dns/node-resolver-dwfmh"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284232    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-cnibin\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284214    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cedda70a-9baa-4592-818a-63092c854c8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284260    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284278    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-etc-selinux\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284292    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-lib-modules\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284291    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7a73dcc-6a78-4344-b0b3-7a8611466a41-original-pull-secret\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284309    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-run-netns\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284337    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a675f8c3-6d56-485d-8c87-438d66ebdc1e-tmp-dir\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284356    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6035c60e-c58e-455e-a61f-19eb599272af-etc-selinux\") pod \"aws-ebs-csi-driver-node-7nmjp\" (UID: \"6035c60e-c58e-455e-a61f-19eb599272af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nmjp"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284387    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-host-run-netns\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284422    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-lib-modules\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284358    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6b01178-0262-47a5-81bb-d715f5cf4380-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284448    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a675f8c3-6d56-485d-8c87-438d66ebdc1e-tmp-dir\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284464    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cedda70a-9baa-4592-818a-63092c854c8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284466    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-tls\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284504    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-tuning-conf-dir\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284528    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysctl-d\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284542    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284552    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35333d53-b864-432d-af6d-b1baec362af6-serviceca\") pod \"node-ca-bxxz2\" (UID: \"35333d53-b864-432d-af6d-b1baec362af6\") " pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284587    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6b01178-0262-47a5-81bb-d715f5cf4380-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284602    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-tuning-conf-dir\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284614    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xn9h\" (UniqueName: \"kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-kube-api-access-7xn9h\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284615    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6b01178-0262-47a5-81bb-d715f5cf4380-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284641    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-system-cni-dir\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284665    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-tls\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284669    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-netns\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284708    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee3c157f-683e-415d-b856-74da8b4fe669-system-cni-dir\") pod \"multus-additional-cni-plugins-64whk\" (UID: \"ee3c157f-683e-415d-b856-74da8b4fe669\") " pod="openshift-multus/multus-additional-cni-plugins-64whk"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284755    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6b01178-0262-47a5-81bb-d715f5cf4380-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284747    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-netns\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284784    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysctl-d\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284784    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a675f8c3-6d56-485d-8c87-438d66ebdc1e-metrics-tls\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284817    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysconfig\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284838    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-host\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284860    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284892    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-accelerators-collector-config\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284896    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysconfig\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284941    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-run-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.284950    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-host\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285008    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a675f8c3-6d56-485d-8c87-438d66ebdc1e-metrics-tls\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285062    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl4qt\" (UniqueName: \"kubernetes.io/projected/92d09c10-51ac-4231-8195-d0c853887162-kube-api-access-cl4qt\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285089    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-accelerators-collector-config\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285091    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-var-lib-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285123    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-var-lib-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285134    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4201a324-4bf9-46fc-b1be-c8381f3e9dba-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285159    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-root\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285186    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-etc-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285193    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35333d53-b864-432d-af6d-b1baec362af6-serviceca\") pod \"node-ca-bxxz2\" (UID: \"35333d53-b864-432d-af6d-b1baec362af6\") " pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285209    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t74h\" (UniqueName: \"kubernetes.io/projected/cedda70a-9baa-4592-818a-63092c854c8e-kube-api-access-4t74h\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285259    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-config-volume\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285289    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-web-config\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285314    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-k8s-cni-cncf-io\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285338    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysctl-conf\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285366    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-var-lib-kubelet\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285391    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-systemd-units\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285418    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4jj8\" (UniqueName: \"kubernetes.io/projected/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-api-access-g4jj8\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285442    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-config-volume\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285259    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4201a324-4bf9-46fc-b1be-c8381f3e9dba-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285463    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-etc-openvswitch\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285445    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-conf-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285503    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285515    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-var-lib-kubelet\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285567    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5aad5010-0827-4918-85d9-47e337a22774-systemd-units\") pod \"ovnkube-node-7rv7g\" (UID: \"5aad5010-0827-4918-85d9-47e337a22774\") " pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285649    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-web-config\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285692    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-multus-conf-dir\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285706    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285734    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cdfe62e2-9594-4c48-a3bc-2dce9df78b56-host-run-k8s-cni-cncf-io\") pod \"multus-vjc5d\" (UID: \"cdfe62e2-9594-4c48-a3bc-2dce9df78b56\") " pod="openshift-multus/multus-vjc5d"
Oct 30 19:32:02.286166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.285857    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b16dc00a-c3b0-4767-a468-6d204ba44e55-etc-sysctl-conf\") pod \"tuned-4pwl6\" (UID: \"b16dc00a-c3b0-4767-a468-6d204ba44e55\") " pod="openshift-cluster-node-tuning-operator/tuned-4pwl6"
Oct 30 19:32:02.296334 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.296153    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwlx\" (UniqueName: \"kubernetes.io/projected/35333d53-b864-432d-af6d-b1baec362af6-kube-api-access-mbwlx\") pod \"node-ca-bxxz2\" (UID: \"35333d53-b864-432d-af6d-b1baec362af6\") " pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:32:02.296334 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.296245    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqggx\" (UniqueName: \"kubernetes.io/projected/a675f8c3-6d56-485d-8c87-438d66ebdc1e-kube-api-access-xqggx\") pod \"dns-default-pzxdj\" (UID: \"a675f8c3-6d56-485d-8c87-438d66ebdc1e\") " pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:32:02.296334 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.296291    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t74h\" (UniqueName: \"kubernetes.io/projected/cedda70a-9baa-4592-818a-63092c854c8e-kube-api-access-4t74h\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:32:02.296334 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.296318    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vpzr\" (UniqueName: \"kubernetes.io/projected/cfda46fc-34fd-461d-a2bf-d4ec19596ed2-kube-api-access-8vpzr\") pod \"ingress-canary-crfs6\" (UID: \"cfda46fc-34fd-461d-a2bf-d4ec19596ed2\") " pod="openshift-ingress-canary/ingress-canary-crfs6"
Oct 30 19:32:02.296585 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.296447    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvm7j\" (UniqueName: \"kubernetes.io/projected/f6b01178-0262-47a5-81bb-d715f5cf4380-kube-api-access-bvm7j\") pod \"prometheus-operator-9cd89b74c-wwrm5\" (UID: \"f6b01178-0262-47a5-81bb-d715f5cf4380\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wwrm5"
Oct 30 19:32:02.296585 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.296509    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl4qt\" (UniqueName: \"kubernetes.io/projected/92d09c10-51ac-4231-8195-d0c853887162-kube-api-access-cl4qt\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.296924 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.296902    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4jj8\" (UniqueName: \"kubernetes.io/projected/4201a324-4bf9-46fc-b1be-c8381f3e9dba-kube-api-access-g4jj8\") pod \"kube-state-metrics-5dc8d7d89b-cfmcp\" (UID: \"4201a324-4bf9-46fc-b1be-c8381f3e9dba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfmcp"
Oct 30 19:32:02.296969 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.296955    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xn9h\" (UniqueName: \"kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-kube-api-access-7xn9h\") pod \"alertmanager-main-0\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:02.385932 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.385887    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7a73dcc-6a78-4344-b0b3-7a8611466a41-kubelet-config\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:32:02.385932 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.385933    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35333d53-b864-432d-af6d-b1baec362af6-host\") pod \"node-ca-bxxz2\" (UID: \"35333d53-b864-432d-af6d-b1baec362af6\") " pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:32:02.386194 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.385975    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-root\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.386194 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.386001    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cedda70a-9baa-4592-818a-63092c854c8e-crio-socket\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:32:02.386194 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.386018    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7a73dcc-6a78-4344-b0b3-7a8611466a41-dbus\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:32:02.386194 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.386066    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-sys\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.386194 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.386069    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-root\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.386194 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.386093    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-wtmp\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.386194 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.386017    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7a73dcc-6a78-4344-b0b3-7a8611466a41-kubelet-config\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:32:02.386194 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.386112    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cedda70a-9baa-4592-818a-63092c854c8e-crio-socket\") pod \"insights-runtime-extractor-xbxhg\" (UID: \"cedda70a-9baa-4592-818a-63092c854c8e\") " pod="openshift-insights/insights-runtime-extractor-xbxhg"
Oct 30 19:32:02.386194 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.386129    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35333d53-b864-432d-af6d-b1baec362af6-host\") pod \"node-ca-bxxz2\" (UID: \"35333d53-b864-432d-af6d-b1baec362af6\") " pod="openshift-image-registry/node-ca-bxxz2"
Oct 30 19:32:02.386194 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.386131    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7a73dcc-6a78-4344-b0b3-7a8611466a41-dbus\") pod \"global-pull-secret-syncer-6vm8t\" (UID: \"b7a73dcc-6a78-4344-b0b3-7a8611466a41\") " pod="kube-system/global-pull-secret-syncer-6vm8t"
Oct 30 19:32:02.386194 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.386156    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-node-exporter-wtmp\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:02.386194 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:02.386158    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92d09c10-51ac-4231-8195-d0c853887162-sys\") pod \"node-exporter-4jpzb\" (UID: \"92d09c10-51ac-4231-8195-d0c853887162\") " pod="openshift-monitoring/node-exporter-4jpzb"
Oct 30 19:32:04.286595 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:32:04.286556    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:32:05.088470 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:05.088436    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pzxdj"
Oct 30 19:32:05.122184 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:05.122157    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:06.555443 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:06.555407    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:32:08.128484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:08.128453    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qnmn2"
Oct 30 19:32:10.097066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:10.097028    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:10.097458 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:10.097267    6030 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:32:23.146111 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.146038    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:32:23.146562 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.146484    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="alertmanager" containerID="cri-o://541b595e2bbf504fb8b33ea5d448e2c4a600f864e80eff041795a075edb0e888" gracePeriod=120
Oct 30 19:32:23.146562 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.146517    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="kube-rbac-proxy-metric" containerID="cri-o://5c50947ee1afb5392f692fc6554151edf7227c841ebadc5966b986781edd8fe0" gracePeriod=120
Oct 30 19:32:23.146679 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.146537    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="prom-label-proxy" containerID="cri-o://7448b70599719707de120df46e9e4207a40924bcede75e7e82d7cbadec9c038a" gracePeriod=120
Oct 30 19:32:23.146679 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.146607    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="config-reloader" containerID="cri-o://72126c790b8a7908e6a37fb12dae90c4cf6a449e22e82b63c8a66a064e3a9a6d" gracePeriod=120
Oct 30 19:32:23.146679 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.146572    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="kube-rbac-proxy-web" containerID="cri-o://43abd83f9dcccde3a0c795a13ee94cfdbcaeb7f93e482b59abe9819b5fefe9c9" gracePeriod=120
Oct 30 19:32:23.146830 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.146601    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="kube-rbac-proxy" containerID="cri-o://70a9a42c7f2556f37fcb44d62617400e3daee39e31e4e97e2f40d141a0b1f2ee" gracePeriod=120
Oct 30 19:32:23.323562 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.323533    6030 generic.go:356] "Generic (PLEG): container finished" podID="87f098c7-fd9e-4ca8-85ef-412371228817" containerID="7448b70599719707de120df46e9e4207a40924bcede75e7e82d7cbadec9c038a" exitCode=0
Oct 30 19:32:23.323562 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.323562    6030 generic.go:356] "Generic (PLEG): container finished" podID="87f098c7-fd9e-4ca8-85ef-412371228817" containerID="5c50947ee1afb5392f692fc6554151edf7227c841ebadc5966b986781edd8fe0" exitCode=0
Oct 30 19:32:23.323727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.323568    6030 generic.go:356] "Generic (PLEG): container finished" podID="87f098c7-fd9e-4ca8-85ef-412371228817" containerID="70a9a42c7f2556f37fcb44d62617400e3daee39e31e4e97e2f40d141a0b1f2ee" exitCode=0
Oct 30 19:32:23.323727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.323575    6030 generic.go:356] "Generic (PLEG): container finished" podID="87f098c7-fd9e-4ca8-85ef-412371228817" containerID="43abd83f9dcccde3a0c795a13ee94cfdbcaeb7f93e482b59abe9819b5fefe9c9" exitCode=0
Oct 30 19:32:23.323727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.323580    6030 generic.go:356] "Generic (PLEG): container finished" podID="87f098c7-fd9e-4ca8-85ef-412371228817" containerID="72126c790b8a7908e6a37fb12dae90c4cf6a449e22e82b63c8a66a064e3a9a6d" exitCode=0
Oct 30 19:32:23.323727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.323585    6030 generic.go:356] "Generic (PLEG): container finished" podID="87f098c7-fd9e-4ca8-85ef-412371228817" containerID="541b595e2bbf504fb8b33ea5d448e2c4a600f864e80eff041795a075edb0e888" exitCode=0
Oct 30 19:32:23.323727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.323628    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"87f098c7-fd9e-4ca8-85ef-412371228817","Type":"ContainerDied","Data":"7448b70599719707de120df46e9e4207a40924bcede75e7e82d7cbadec9c038a"}
Oct 30 19:32:23.323727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.323704    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"87f098c7-fd9e-4ca8-85ef-412371228817","Type":"ContainerDied","Data":"5c50947ee1afb5392f692fc6554151edf7227c841ebadc5966b986781edd8fe0"}
Oct 30 19:32:23.323932 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.323732    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"87f098c7-fd9e-4ca8-85ef-412371228817","Type":"ContainerDied","Data":"70a9a42c7f2556f37fcb44d62617400e3daee39e31e4e97e2f40d141a0b1f2ee"}
Oct 30 19:32:23.323932 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.323755    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"87f098c7-fd9e-4ca8-85ef-412371228817","Type":"ContainerDied","Data":"43abd83f9dcccde3a0c795a13ee94cfdbcaeb7f93e482b59abe9819b5fefe9c9"}
Oct 30 19:32:23.323932 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.323777    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"87f098c7-fd9e-4ca8-85ef-412371228817","Type":"ContainerDied","Data":"72126c790b8a7908e6a37fb12dae90c4cf6a449e22e82b63c8a66a064e3a9a6d"}
Oct 30 19:32:23.323932 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.323800    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"87f098c7-fd9e-4ca8-85ef-412371228817","Type":"ContainerDied","Data":"541b595e2bbf504fb8b33ea5d448e2c4a600f864e80eff041795a075edb0e888"}
Oct 30 19:32:23.390422 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.390400    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:23.517005 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.516921    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-trusted-ca-bundle\") pod \"87f098c7-fd9e-4ca8-85ef-412371228817\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") "
Oct 30 19:32:23.517005 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.516969    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-metrics-client-ca\") pod \"87f098c7-fd9e-4ca8-85ef-412371228817\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") "
Oct 30 19:32:23.517005 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.516998    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-web-config\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-web-config\") pod \"87f098c7-fd9e-4ca8-85ef-412371228817\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") "
Oct 30 19:32:23.517310 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.517020    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-config-out\" (UniqueName: \"kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-config-out\") pod \"87f098c7-fd9e-4ca8-85ef-412371228817\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") "
Oct 30 19:32:23.517310 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.517036    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-main-db\") pod \"87f098c7-fd9e-4ca8-85ef-412371228817\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") "
Oct 30 19:32:23.517310 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.517085    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy\") pod \"87f098c7-fd9e-4ca8-85ef-412371228817\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") "
Oct 30 19:32:23.517310 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.517108    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-web\") pod \"87f098c7-fd9e-4ca8-85ef-412371228817\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") "
Oct 30 19:32:23.517310 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.517132    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-kube-api-access-7xn9h\" (UniqueName: \"kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-kube-api-access-7xn9h\") pod \"87f098c7-fd9e-4ca8-85ef-412371228817\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") "
Oct 30 19:32:23.517310 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.517163    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-main-tls\") pod \"87f098c7-fd9e-4ca8-85ef-412371228817\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") "
Oct 30 19:32:23.517310 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.517210    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-cluster-tls-config\") pod \"87f098c7-fd9e-4ca8-85ef-412371228817\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") "
Oct 30 19:32:23.517310 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.517249    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-tls-assets\" (UniqueName: \"kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-tls-assets\") pod \"87f098c7-fd9e-4ca8-85ef-412371228817\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") "
Oct 30 19:32:23.517310 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.517289    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-metric\") pod \"87f098c7-fd9e-4ca8-85ef-412371228817\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") "
Oct 30 19:32:23.517737 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.517323    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-config-volume\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-config-volume\") pod \"87f098c7-fd9e-4ca8-85ef-412371228817\" (UID: \"87f098c7-fd9e-4ca8-85ef-412371228817\") "
Oct 30 19:32:23.517737 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.517409    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-trusted-ca-bundle" pod "87f098c7-fd9e-4ca8-85ef-412371228817" (UID: "87f098c7-fd9e-4ca8-85ef-412371228817"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:23.517737 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.517427    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-main-db" pod "87f098c7-fd9e-4ca8-85ef-412371228817" (UID: "87f098c7-fd9e-4ca8-85ef-412371228817"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:23.517737 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.517571    6030 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:23.517737 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.517593    6030 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-alertmanager-main-db\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:23.517737 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.517626    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-metrics-client-ca" pod "87f098c7-fd9e-4ca8-85ef-412371228817" (UID: "87f098c7-fd9e-4ca8-85ef-412371228817"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:23.520784 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.520750    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-web" pod "87f098c7-fd9e-4ca8-85ef-412371228817" (UID: "87f098c7-fd9e-4ca8-85ef-412371228817"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:23.521262 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.521217    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-kube-api-access-7xn9h" pod "87f098c7-fd9e-4ca8-85ef-412371228817" (UID: "87f098c7-fd9e-4ca8-85ef-412371228817"). InnerVolumeSpecName "kube-api-access-7xn9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:23.521368 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.521223    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy" pod "87f098c7-fd9e-4ca8-85ef-412371228817" (UID: "87f098c7-fd9e-4ca8-85ef-412371228817"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:23.521368 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.521271    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-config-out" pod "87f098c7-fd9e-4ca8-85ef-412371228817" (UID: "87f098c7-fd9e-4ca8-85ef-412371228817"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:23.521368 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.521302    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-config-volume" pod "87f098c7-fd9e-4ca8-85ef-412371228817" (UID: "87f098c7-fd9e-4ca8-85ef-412371228817"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:23.521549 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.521396    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-metric" pod "87f098c7-fd9e-4ca8-85ef-412371228817" (UID: "87f098c7-fd9e-4ca8-85ef-412371228817"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:23.521549 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.521471    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-tls-assets" pod "87f098c7-fd9e-4ca8-85ef-412371228817" (UID: "87f098c7-fd9e-4ca8-85ef-412371228817"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:23.521837 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.521820    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-main-tls" pod "87f098c7-fd9e-4ca8-85ef-412371228817" (UID: "87f098c7-fd9e-4ca8-85ef-412371228817"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:23.524887 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.524865    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-cluster-tls-config" pod "87f098c7-fd9e-4ca8-85ef-412371228817" (UID: "87f098c7-fd9e-4ca8-85ef-412371228817"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:23.530358 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.530339    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-web-config" pod "87f098c7-fd9e-4ca8-85ef-412371228817" (UID: "87f098c7-fd9e-4ca8-85ef-412371228817"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:23.618822 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.618785    6030 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/87f098c7-fd9e-4ca8-85ef-412371228817-config-out\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:23.618822 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.618816    6030 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:23.618822 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.618827    6030 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:23.619105 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.618837    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xn9h\" (UniqueName: \"kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-kube-api-access-7xn9h\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:23.619105 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.618847    6030 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-main-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:23.619105 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.618857    6030 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-cluster-tls-config\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:23.619105 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.618866    6030 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/87f098c7-fd9e-4ca8-85ef-412371228817-tls-assets\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:23.619105 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.618874    6030 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:23.619105 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.618883    6030 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:23.619105 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.618892    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87f098c7-fd9e-4ca8-85ef-412371228817-metrics-client-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:23.619105 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:23.618900    6030 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/87f098c7-fd9e-4ca8-85ef-412371228817-web-config\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:24.330100 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.330067    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"87f098c7-fd9e-4ca8-85ef-412371228817","Type":"ContainerDied","Data":"d7f07af5464a3d7f7b8c28b7ca81681ec36e68037aa347554caf31757796be2c"}
Oct 30 19:32:24.330469 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.330134    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.330469 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.330141    6030 scope.go:122] "RemoveContainer" containerID="7448b70599719707de120df46e9e4207a40924bcede75e7e82d7cbadec9c038a"
Oct 30 19:32:24.337945 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.337924    6030 scope.go:122] "RemoveContainer" containerID="5c50947ee1afb5392f692fc6554151edf7227c841ebadc5966b986781edd8fe0"
Oct 30 19:32:24.345644 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.345620    6030 scope.go:122] "RemoveContainer" containerID="70a9a42c7f2556f37fcb44d62617400e3daee39e31e4e97e2f40d141a0b1f2ee"
Oct 30 19:32:24.353489 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.353463    6030 scope.go:122] "RemoveContainer" containerID="43abd83f9dcccde3a0c795a13ee94cfdbcaeb7f93e482b59abe9819b5fefe9c9"
Oct 30 19:32:24.358866 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.358838    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:32:24.360645 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.360628    6030 scope.go:122] "RemoveContainer" containerID="72126c790b8a7908e6a37fb12dae90c4cf6a449e22e82b63c8a66a064e3a9a6d"
Oct 30 19:32:24.367929 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.367911    6030 scope.go:122] "RemoveContainer" containerID="541b595e2bbf504fb8b33ea5d448e2c4a600f864e80eff041795a075edb0e888"
Oct 30 19:32:24.371967 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.371947    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:32:24.375524 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.375504    6030 scope.go:122] "RemoveContainer" containerID="3de25613b3637d81768b30d3e67b2c05d438c51ccf34e834f06ad459f38dbd6c"
Oct 30 19:32:24.410162 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410102    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:32:24.410347 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410335    6030 cpu_manager.go:414] "containerMap: removing container" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="init-config-reloader"
Oct 30 19:32:24.410392 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410349    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="init-config-reloader"
Oct 30 19:32:24.410392 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410361    6030 cpu_manager.go:414] "containerMap: removing container" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="kube-rbac-proxy-metric"
Oct 30 19:32:24.410392 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410366    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="kube-rbac-proxy-metric"
Oct 30 19:32:24.410392 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410374    6030 cpu_manager.go:414] "containerMap: removing container" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="prom-label-proxy"
Oct 30 19:32:24.410392 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410379    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="prom-label-proxy"
Oct 30 19:32:24.410392 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410389    6030 cpu_manager.go:414] "containerMap: removing container" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="kube-rbac-proxy-web"
Oct 30 19:32:24.410392 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410393    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="kube-rbac-proxy-web"
Oct 30 19:32:24.410619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410399    6030 cpu_manager.go:414] "containerMap: removing container" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="alertmanager"
Oct 30 19:32:24.410619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410404    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="alertmanager"
Oct 30 19:32:24.410619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410416    6030 cpu_manager.go:414] "containerMap: removing container" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="kube-rbac-proxy"
Oct 30 19:32:24.410619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410421    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="kube-rbac-proxy"
Oct 30 19:32:24.410619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410426    6030 cpu_manager.go:414] "containerMap: removing container" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="config-reloader"
Oct 30 19:32:24.410619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410430    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="config-reloader"
Oct 30 19:32:24.410619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410469    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="kube-rbac-proxy-web"
Oct 30 19:32:24.410619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410475    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="alertmanager"
Oct 30 19:32:24.410619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410482    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="kube-rbac-proxy"
Oct 30 19:32:24.410619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410487    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="config-reloader"
Oct 30 19:32:24.410619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410493    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="init-config-reloader"
Oct 30 19:32:24.410619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410500    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="kube-rbac-proxy-metric"
Oct 30 19:32:24.410619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.410505    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" containerName="prom-label-proxy"
Oct 30 19:32:24.415397 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.415380    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.426808 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.426788    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 19:32:24.426898 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.426843    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 19:32:24.426898 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.426886    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 19:32:24.427003 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.426916    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 19:32:24.427124 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.427108    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 19:32:24.427324 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.427307    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 19:32:24.427426 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.427349    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 19:32:24.427426 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.427358    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hwkcq\""
Oct 30 19:32:24.427426 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.427388    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 19:32:24.432652 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.432630    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 19:32:24.433469 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.433455    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:32:24.524670 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.524633    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.524851 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.524684    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-config-out\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.524851 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.524711    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.524851 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.524741    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.524851 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.524802    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.524851 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.524846    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.525005 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.524905    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.525005 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.524952    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.525005 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.524972    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-web-config\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.525005 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.524998    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.525175 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.525018    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5bs6\" (UniqueName: \"kubernetes.io/projected/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-kube-api-access-n5bs6\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.525175 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.525135    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-config-volume\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.525247 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.525168    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.625950 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.625916    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.625950 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.625960    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5bs6\" (UniqueName: \"kubernetes.io/projected/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-kube-api-access-n5bs6\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.626235 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.626009    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-config-volume\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.626235 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.626064    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.626235 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.626093    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.626235 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.626124    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-config-out\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.626235 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.626147    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.626235 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.626181    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.626235 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.626207    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.626579 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.626241    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.626579 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.626265    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.626579 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.626295    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.626579 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.626318    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-web-config\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.626579 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.626368    6030 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:32:24.628700 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.627663    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.628700 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.627917    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.628700 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.628533    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.629480 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.629323    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.629480 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.629476    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.629658 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.629522    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.629658 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.629567    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-web-config\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.629658 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.629615    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.630039 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.629692    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.630435 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.630416    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.631167 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.631146    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-config-volume\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.631300 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.631283    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-config-out\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.639218 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.639196    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5bs6\" (UniqueName: \"kubernetes.io/projected/96d833e9-2afc-45f9-a0aa-3c06cb5795b1-kube-api-access-n5bs6\") pod \"alertmanager-main-0\" (UID: \"96d833e9-2afc-45f9-a0aa-3c06cb5795b1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:24.742571 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:24.742483    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:32:25.026205 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:25.026118    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:32:25.029658 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:32:25.029621    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d833e9_2afc_45f9_a0aa_3c06cb5795b1.slice/crio-1951aed94049120704590d45ae83388b5110397d4eb0be250dee55587eec2916 WatchSource:0}: Error finding container 1951aed94049120704590d45ae83388b5110397d4eb0be250dee55587eec2916: Status 404 returned error can't find the container with id 1951aed94049120704590d45ae83388b5110397d4eb0be250dee55587eec2916
Oct 30 19:32:25.200329 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:25.200298    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87f098c7-fd9e-4ca8-85ef-412371228817" path="/var/lib/kubelet/pods/87f098c7-fd9e-4ca8-85ef-412371228817/volumes"
Oct 30 19:32:25.336198 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:25.336113    6030 generic.go:356] "Generic (PLEG): container finished" podID="96d833e9-2afc-45f9-a0aa-3c06cb5795b1" containerID="0d1153555961e736939ad8cd2a5c642be82ec0885d8de6dce1c3301838b005b4" exitCode=0
Oct 30 19:32:25.336568 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:25.336200    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96d833e9-2afc-45f9-a0aa-3c06cb5795b1","Type":"ContainerDied","Data":"0d1153555961e736939ad8cd2a5c642be82ec0885d8de6dce1c3301838b005b4"}
Oct 30 19:32:25.336568 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:25.336272    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96d833e9-2afc-45f9-a0aa-3c06cb5795b1","Type":"ContainerStarted","Data":"1951aed94049120704590d45ae83388b5110397d4eb0be250dee55587eec2916"}
Oct 30 19:32:26.342705 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:26.342667    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96d833e9-2afc-45f9-a0aa-3c06cb5795b1","Type":"ContainerStarted","Data":"ca3387865e754535191cda97648aceb7b272581ebf331b7c0169c18d60b9f62c"}
Oct 30 19:32:26.343257 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:26.342747    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96d833e9-2afc-45f9-a0aa-3c06cb5795b1","Type":"ContainerStarted","Data":"0ee8adf06ad697c1bd6028282f0013421ddddf432c787441cd9aa7ace18d45b1"}
Oct 30 19:32:26.343257 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:26.342801    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96d833e9-2afc-45f9-a0aa-3c06cb5795b1","Type":"ContainerStarted","Data":"92e71dbd7823142c90c079635f2a64e14477cc9f34d3007b8735600cc39094c0"}
Oct 30 19:32:26.343257 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:26.342847    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96d833e9-2afc-45f9-a0aa-3c06cb5795b1","Type":"ContainerStarted","Data":"ec6e6a1d16205f74df429d001eb16edbdfecca28e5e5961104d3a70240e6ec1c"}
Oct 30 19:32:26.343257 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:26.342898    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96d833e9-2afc-45f9-a0aa-3c06cb5795b1","Type":"ContainerStarted","Data":"c0638a903c3ab1499034016059df20e11579fba00e6f826fa3eaeae3b5766e05"}
Oct 30 19:32:26.343257 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:26.342932    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96d833e9-2afc-45f9-a0aa-3c06cb5795b1","Type":"ContainerStarted","Data":"d263d2d2e1ea5ac1547dec9d07dc081fe1ccfb087d7c7543831659f440bed5d6"}
Oct 30 19:32:26.381661 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:26.381604    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.369431447 podStartE2EDuration="2.381590909s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.159462ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:26.381394408 +0000 UTC m=+25.297001265" watchObservedRunningTime="2026-10-30 19:32:26.381590909 +0000 UTC m=+25.297197759"
Oct 30 19:32:31.932260 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:31.932094    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7rv7g"
Oct 30 19:32:34.286878 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:32:34.286843    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:32:36.558757 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:32:36.558729    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hr4sv"
Oct 30 19:33:04.287118 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:33:04.287080    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:33:34.286426 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:33:34.286377    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:34:04.285348 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:34:04.285253    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:34:34.287278 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:34:34.287240    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:34:41.858785 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.858753    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9"]
Oct 30 19:34:41.862022 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.861996    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9"
Oct 30 19:34:41.869882 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.869859    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-pgm5r\""
Oct 30 19:34:41.872631 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.872605    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 19:34:41.872737 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.872721    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 19:34:41.947868 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.947839    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"]
Oct 30 19:34:41.951309 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.951278    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9"]
Oct 30 19:34:41.951452 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.951351    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:41.970120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.970083    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 19:34:41.970284 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.970140    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 19:34:41.970284 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.970146    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 19:34:41.970432 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.970417    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:34:41.970484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.970422    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:34:41.970484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.970438    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:34:41.970484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.970451    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gmjw8\""
Oct 30 19:34:41.974078 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.974030    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"]
Oct 30 19:34:41.989706 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.989681    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9\" (UID: \"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9"
Oct 30 19:34:41.989850 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.989715    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n6qv\" (UniqueName: \"kubernetes.io/projected/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-kube-api-access-6n6qv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9\" (UID: \"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9"
Oct 30 19:34:41.989850 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:41.989737    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9\" (UID: \"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9"
Oct 30 19:34:42.091130 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.091095    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtwrh\" (UniqueName: \"kubernetes.io/projected/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-kube-api-access-xtwrh\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.091294 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.091142    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.091294 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.091229    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.091294 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.091272    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.091397 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.091341    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9\" (UID: \"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9"
Oct 30 19:34:42.091447 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.091427    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n6qv\" (UniqueName: \"kubernetes.io/projected/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-kube-api-access-6n6qv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9\" (UID: \"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9"
Oct 30 19:34:42.091484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.091457    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.091484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.091479    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.091561 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.091502    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9\" (UID: \"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9"
Oct 30 19:34:42.091691 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.091667    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.091691 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.091677    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9\" (UID: \"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9"
Oct 30 19:34:42.091812 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.091796    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9\" (UID: \"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9"
Oct 30 19:34:42.106233 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.106204    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n6qv\" (UniqueName: \"kubernetes.io/projected/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-kube-api-access-6n6qv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9\" (UID: \"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9"
Oct 30 19:34:42.178354 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.178322    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9"
Oct 30 19:34:42.192636 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.192607    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.192764 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.192641    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.192764 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.192667    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.192873 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.192795    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtwrh\" (UniqueName: \"kubernetes.io/projected/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-kube-api-access-xtwrh\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.192873 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.192845    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.192971 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.192879    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.194073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.193602    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.194073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.193845    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.199574 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.195925    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.199574 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.196375    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.201372 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.201337    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.201485 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.201450    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.206975 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.206885    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtwrh\" (UniqueName: \"kubernetes.io/projected/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-kube-api-access-xtwrh\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.209175 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.209153    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-s74hk\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.267535 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.267501    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:42.458624 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.458543    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9"]
Oct 30 19:34:42.461520 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:34:42.461493    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c770b5a_b07f_43f0_bbe0_a3939e65e2d8.slice/crio-1be638a0ec2fc2c76c9d1a921157a8c786f01bc4a66229c68fadce5711ad82ee WatchSource:0}: Error finding container 1be638a0ec2fc2c76c9d1a921157a8c786f01bc4a66229c68fadce5711ad82ee: Status 404 returned error can't find the container with id 1be638a0ec2fc2c76c9d1a921157a8c786f01bc4a66229c68fadce5711ad82ee
Oct 30 19:34:42.463462 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.463443    6030 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:34:42.553406 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.553375    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"]
Oct 30 19:34:42.556465 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:34:42.556441    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceaabe36_9009_4d0a_9a0a_c48b23bfd93b.slice/crio-72ff1b6566d9b7e2f872a74c18fb7add20062f2b8d9635da338561571aeceb9a WatchSource:0}: Error finding container 72ff1b6566d9b7e2f872a74c18fb7add20062f2b8d9635da338561571aeceb9a: Status 404 returned error can't find the container with id 72ff1b6566d9b7e2f872a74c18fb7add20062f2b8d9635da338561571aeceb9a
Oct 30 19:34:42.772001 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.771913    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9" event={"ID":"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8","Type":"ContainerStarted","Data":"1be638a0ec2fc2c76c9d1a921157a8c786f01bc4a66229c68fadce5711ad82ee"}
Oct 30 19:34:42.773133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:42.773109    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk" event={"ID":"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b","Type":"ContainerStarted","Data":"72ff1b6566d9b7e2f872a74c18fb7add20062f2b8d9635da338561571aeceb9a"}
Oct 30 19:34:45.748320 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:45.748283    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:34:45.748744 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:45.748369    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:34:46.791904 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:46.791869    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk" event={"ID":"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b","Type":"ContainerStarted","Data":"16f3b83751bef482f5b34ab95c562a9cb93147ba2cf79c1e1941b90ebd4608d3"}
Oct 30 19:34:46.792436 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:46.792140    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:46.793990 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:46.793913    6030 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-s74hk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:34:46.793990 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:46.793972    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk" podUID="ceaabe36-9009-4d0a-9a0a-c48b23bfd93b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:34:47.796809 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:47.796780    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:34:47.828923 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:47.828867    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk" podStartSLOduration=3.639918206 podStartE2EDuration="6.828850633s" totalImagesPullingTime="3.188932427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:46.824904458 +0000 UTC m=+165.740511314" watchObservedRunningTime="2026-10-30 19:34:47.828850633 +0000 UTC m=+166.744457490"
Oct 30 19:34:48.801153 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:48.801118    6030 generic.go:356] "Generic (PLEG): container finished" podID="4c770b5a-b07f-43f0-bbe0-a3939e65e2d8" containerID="f18f9006483830a5cbb9cc1553f35fafd4b3b5ab22bf0935d741ec43bfe80235" exitCode=0
Oct 30 19:34:48.801510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:48.801211    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9" event={"ID":"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8","Type":"ContainerDied","Data":"f18f9006483830a5cbb9cc1553f35fafd4b3b5ab22bf0935d741ec43bfe80235"}
Oct 30 19:34:49.806247 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:49.806207    6030 generic.go:356] "Generic (PLEG): container finished" podID="4c770b5a-b07f-43f0-bbe0-a3939e65e2d8" containerID="b1e03a1b78f0f945ed0b9dc1c6e0c77899719eac52a57805b6175ad53324d137" exitCode=0
Oct 30 19:34:49.806635 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:49.806288    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9" event={"ID":"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8","Type":"ContainerDied","Data":"b1e03a1b78f0f945ed0b9dc1c6e0c77899719eac52a57805b6175ad53324d137"}
Oct 30 19:34:55.830609 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:55.830531    6030 generic.go:356] "Generic (PLEG): container finished" podID="4c770b5a-b07f-43f0-bbe0-a3939e65e2d8" containerID="a779d980e58ea03c1ee28fc874a4656ba7fb111815e023dd8904f96b4d888e18" exitCode=0
Oct 30 19:34:55.830609 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:55.830568    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9" event={"ID":"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8","Type":"ContainerDied","Data":"a779d980e58ea03c1ee28fc874a4656ba7fb111815e023dd8904f96b4d888e18"}
Oct 30 19:34:56.956997 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:56.956975    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9"
Oct 30 19:34:57.017699 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:57.017667    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-util\" (UniqueName: \"kubernetes.io/empty-dir/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-util\") pod \"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8\" (UID: \"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8\") "
Oct 30 19:34:57.017880 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:57.017771    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-kube-api-access-6n6qv\" (UniqueName: \"kubernetes.io/projected/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-kube-api-access-6n6qv\") pod \"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8\" (UID: \"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8\") "
Oct 30 19:34:57.017880 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:57.017811    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-bundle\") pod \"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8\" (UID: \"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8\") "
Oct 30 19:34:57.018547 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:57.018463    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-bundle" pod "4c770b5a-b07f-43f0-bbe0-a3939e65e2d8" (UID: "4c770b5a-b07f-43f0-bbe0-a3939e65e2d8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:57.020114 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:57.020089    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-kube-api-access-6n6qv" pod "4c770b5a-b07f-43f0-bbe0-a3939e65e2d8" (UID: "4c770b5a-b07f-43f0-bbe0-a3939e65e2d8"). InnerVolumeSpecName "kube-api-access-6n6qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:57.022282 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:57.022262    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-util" pod "4c770b5a-b07f-43f0-bbe0-a3939e65e2d8" (UID: "4c770b5a-b07f-43f0-bbe0-a3939e65e2d8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:57.118572 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:57.118491    6030 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-bundle\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.118572 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:57.118521    6030 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-util\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.118572 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:57.118530    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n6qv\" (UniqueName: \"kubernetes.io/projected/4c770b5a-b07f-43f0-bbe0-a3939e65e2d8-kube-api-access-6n6qv\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.839265 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:57.839229    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9" event={"ID":"4c770b5a-b07f-43f0-bbe0-a3939e65e2d8","Type":"ContainerDied","Data":"1be638a0ec2fc2c76c9d1a921157a8c786f01bc4a66229c68fadce5711ad82ee"}
Oct 30 19:34:57.839265 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:57.839248    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p6pd9"
Oct 30 19:34:57.839467 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:34:57.839284    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1be638a0ec2fc2c76c9d1a921157a8c786f01bc4a66229c68fadce5711ad82ee"
Oct 30 19:35:04.285967 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:35:04.285928    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:35:29.760808 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.760766    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"]
Oct 30 19:35:29.761790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.761212    6030 cpu_manager.go:414] "containerMap: removing container" podUID="4c770b5a-b07f-43f0-bbe0-a3939e65e2d8" containerName="pull"
Oct 30 19:35:29.761790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.761231    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c770b5a-b07f-43f0-bbe0-a3939e65e2d8" containerName="pull"
Oct 30 19:35:29.761790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.761256    6030 cpu_manager.go:414] "containerMap: removing container" podUID="4c770b5a-b07f-43f0-bbe0-a3939e65e2d8" containerName="extract"
Oct 30 19:35:29.761790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.761266    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c770b5a-b07f-43f0-bbe0-a3939e65e2d8" containerName="extract"
Oct 30 19:35:29.761790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.761286    6030 cpu_manager.go:414] "containerMap: removing container" podUID="4c770b5a-b07f-43f0-bbe0-a3939e65e2d8" containerName="util"
Oct 30 19:35:29.761790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.761294    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c770b5a-b07f-43f0-bbe0-a3939e65e2d8" containerName="util"
Oct 30 19:35:29.761790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.761414    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c770b5a-b07f-43f0-bbe0-a3939e65e2d8" containerName="extract"
Oct 30 19:35:29.763769 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.763721    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.798555 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.798518    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"]
Oct 30 19:35:29.830911 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.830880    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-98qhj\""
Oct 30 19:35:29.831110 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.830951    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 19:35:29.831110 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.830951    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 19:35:29.831110 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.830959    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 19:35:29.831240 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.831115    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 19:35:29.831302 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.831288    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 19:35:29.853642 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.853616    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-metrics-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.853769 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.853648    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jxt9\" (UniqueName: \"kubernetes.io/projected/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-kube-api-access-5jxt9\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.853769 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.853693    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.853851 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.853789    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-webhook-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.853851 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.853838    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-manifests\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.853915 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.853861    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-tmp\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.954363 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.954326    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-webhook-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.954363 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.954364    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-manifests\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.954634 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.954394    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-tmp\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.954634 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.954511    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-metrics-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.954634 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.954558    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jxt9\" (UniqueName: \"kubernetes.io/projected/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-kube-api-access-5jxt9\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.954808 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.954637    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.954864 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.954830    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-manifests\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.957114 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.957090    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-tmp\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.957292 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.957272    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.957416 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.957398    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-metrics-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.957472 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.957456    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-webhook-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:29.972828 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:29.972801    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jxt9\" (UniqueName: \"kubernetes.io/projected/0adacc68-1634-4dd6-9c47-21c2a8a0d79d-kube-api-access-5jxt9\") pod \"opendatahub-operator-controller-manager-78f5b8588d-hthgg\" (UID: \"0adacc68-1634-4dd6-9c47-21c2a8a0d79d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:30.076071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:30.075968    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:30.427329 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:30.427300    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"]
Oct 30 19:35:30.429825 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:35:30.429799    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0adacc68_1634_4dd6_9c47_21c2a8a0d79d.slice/crio-80b7c3a27f33b1d43d62a095fc16cedb4f88303b6cd396f19fbb5c1da81d4985 WatchSource:0}: Error finding container 80b7c3a27f33b1d43d62a095fc16cedb4f88303b6cd396f19fbb5c1da81d4985: Status 404 returned error can't find the container with id 80b7c3a27f33b1d43d62a095fc16cedb4f88303b6cd396f19fbb5c1da81d4985
Oct 30 19:35:30.949361 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:30.949314    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg" event={"ID":"0adacc68-1634-4dd6-9c47-21c2a8a0d79d","Type":"ContainerStarted","Data":"80b7c3a27f33b1d43d62a095fc16cedb4f88303b6cd396f19fbb5c1da81d4985"}
Oct 30 19:35:33.960728 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:33.960650    6030 generic.go:356] "Generic (PLEG): container finished" podID="0adacc68-1634-4dd6-9c47-21c2a8a0d79d" containerID="542c4714ed45df9ac1caa48b6192ab4a828bebc38913cd33126925daa9406b1e" exitCode=0
Oct 30 19:35:33.961110 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:33.960735    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg" event={"ID":"0adacc68-1634-4dd6-9c47-21c2a8a0d79d","Type":"ContainerDied","Data":"542c4714ed45df9ac1caa48b6192ab4a828bebc38913cd33126925daa9406b1e"}
Oct 30 19:35:34.286084 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:35:34.285978    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:35:34.965810 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:34.965772    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg" event={"ID":"0adacc68-1634-4dd6-9c47-21c2a8a0d79d","Type":"ContainerStarted","Data":"7dbad03e07a62084a8a2f9eba4b015f91fa6f31595a894430f1331fc61df5907"}
Oct 30 19:35:34.966195 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:34.965996    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:35.015236 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:35.015169    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg" podStartSLOduration=2.350499181 podStartE2EDuration="6.015155715s" totalImagesPullingTime="3.249368487s" totalInitContainerRuntime="415.288047ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:35.013212653 +0000 UTC m=+213.928819510" watchObservedRunningTime="2026-10-30 19:35:35.015155715 +0000 UTC m=+213.930762571"
Oct 30 19:35:45.972287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:45.972257    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-hthgg"
Oct 30 19:35:47.358645 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:47.358611    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-mgclc"]
Oct 30 19:35:47.360929 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:47.360913    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mgclc"
Oct 30 19:35:47.375756 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:47.375733    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 19:35:47.375867 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:47.375775    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 19:35:47.375867 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:47.375828    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 19:35:47.376181 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:47.376167    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-whmgw\""
Oct 30 19:35:47.382472 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:47.382443    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-mgclc"]
Oct 30 19:35:47.501642 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:47.501610    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffmdl\" (UniqueName: \"kubernetes.io/projected/7d40f8de-854b-4a46-bf24-a37b49e75583-kube-api-access-ffmdl\") pod \"dns-operator-controller-manager-8557cbdb84-mgclc\" (UID: \"7d40f8de-854b-4a46-bf24-a37b49e75583\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mgclc"
Oct 30 19:35:47.602961 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:47.602923    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffmdl\" (UniqueName: \"kubernetes.io/projected/7d40f8de-854b-4a46-bf24-a37b49e75583-kube-api-access-ffmdl\") pod \"dns-operator-controller-manager-8557cbdb84-mgclc\" (UID: \"7d40f8de-854b-4a46-bf24-a37b49e75583\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mgclc"
Oct 30 19:35:47.618744 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:47.618664    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffmdl\" (UniqueName: \"kubernetes.io/projected/7d40f8de-854b-4a46-bf24-a37b49e75583-kube-api-access-ffmdl\") pod \"dns-operator-controller-manager-8557cbdb84-mgclc\" (UID: \"7d40f8de-854b-4a46-bf24-a37b49e75583\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mgclc"
Oct 30 19:35:47.671709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:47.671677    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mgclc"
Oct 30 19:35:47.943903 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:47.943881    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-mgclc"]
Oct 30 19:35:47.946589 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:35:47.946557    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d40f8de_854b_4a46_bf24_a37b49e75583.slice/crio-37e56eed1a55f33c654a3418f743bbf94a37b66018eee4a86ed3c67f6611f469 WatchSource:0}: Error finding container 37e56eed1a55f33c654a3418f743bbf94a37b66018eee4a86ed3c67f6611f469: Status 404 returned error can't find the container with id 37e56eed1a55f33c654a3418f743bbf94a37b66018eee4a86ed3c67f6611f469
Oct 30 19:35:48.010580 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:48.010549    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mgclc" event={"ID":"7d40f8de-854b-4a46-bf24-a37b49e75583","Type":"ContainerStarted","Data":"37e56eed1a55f33c654a3418f743bbf94a37b66018eee4a86ed3c67f6611f469"}
Oct 30 19:35:50.201479 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:50.201448    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6pvgl"]
Oct 30 19:35:50.211585 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:50.211562    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6pvgl"
Oct 30 19:35:50.214521 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:50.214491    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6pvgl"]
Oct 30 19:35:50.220317 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:50.220299    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8hjj4\""
Oct 30 19:35:50.325338 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:50.325299    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgczz\" (UniqueName: \"kubernetes.io/projected/7f8b5c02-97b9-4018-ab0a-43af4476f135-kube-api-access-rgczz\") pod \"limitador-operator-controller-manager-5778bb4449-6pvgl\" (UID: \"7f8b5c02-97b9-4018-ab0a-43af4476f135\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6pvgl"
Oct 30 19:35:50.426816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:50.426773    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgczz\" (UniqueName: \"kubernetes.io/projected/7f8b5c02-97b9-4018-ab0a-43af4476f135-kube-api-access-rgczz\") pod \"limitador-operator-controller-manager-5778bb4449-6pvgl\" (UID: \"7f8b5c02-97b9-4018-ab0a-43af4476f135\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6pvgl"
Oct 30 19:35:50.444849 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:50.444818    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgczz\" (UniqueName: \"kubernetes.io/projected/7f8b5c02-97b9-4018-ab0a-43af4476f135-kube-api-access-rgczz\") pod \"limitador-operator-controller-manager-5778bb4449-6pvgl\" (UID: \"7f8b5c02-97b9-4018-ab0a-43af4476f135\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6pvgl"
Oct 30 19:35:50.529588 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:50.529497    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6pvgl"
Oct 30 19:35:50.828112 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:50.828025    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6pvgl"]
Oct 30 19:35:50.830698 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:35:50.830662    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8b5c02_97b9_4018_ab0a_43af4476f135.slice/crio-46eba3498e473adb76a8ff69dc0ed4745ba2afc18f85caa6d8b2296fa2fe92a5 WatchSource:0}: Error finding container 46eba3498e473adb76a8ff69dc0ed4745ba2afc18f85caa6d8b2296fa2fe92a5: Status 404 returned error can't find the container with id 46eba3498e473adb76a8ff69dc0ed4745ba2afc18f85caa6d8b2296fa2fe92a5
Oct 30 19:35:51.024191 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:51.024150    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mgclc" event={"ID":"7d40f8de-854b-4a46-bf24-a37b49e75583","Type":"ContainerStarted","Data":"f00e8dc0376defb141e25b72c4d94b16fdc4e78f5454b40c69657128ab302743"}
Oct 30 19:35:51.024352 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:51.024327    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mgclc"
Oct 30 19:35:51.025435 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:51.025411    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6pvgl" event={"ID":"7f8b5c02-97b9-4018-ab0a-43af4476f135","Type":"ContainerStarted","Data":"46eba3498e473adb76a8ff69dc0ed4745ba2afc18f85caa6d8b2296fa2fe92a5"}
Oct 30 19:35:51.047416 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:51.047319    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mgclc" podStartSLOduration=1.248695221 podStartE2EDuration="4.047303385s" totalImagesPullingTime="2.798608164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:51.046503586 +0000 UTC m=+229.962110441" watchObservedRunningTime="2026-10-30 19:35:51.047303385 +0000 UTC m=+229.962910243"
Oct 30 19:35:53.034740 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:53.034702    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6pvgl" event={"ID":"7f8b5c02-97b9-4018-ab0a-43af4476f135","Type":"ContainerStarted","Data":"88faf7021f2ac71398d174249a2ad58ae9ab9c48977690290ba5fa0e0ce37c22"}
Oct 30 19:35:53.035180 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:53.034833    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6pvgl"
Oct 30 19:35:53.055442 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:53.055392    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6pvgl" podStartSLOduration=1.484068703 podStartE2EDuration="3.05537831s" totalImagesPullingTime="1.571309607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:53.055004166 +0000 UTC m=+231.970611022" watchObservedRunningTime="2026-10-30 19:35:53.05537831 +0000 UTC m=+231.970985165"
Oct 30 19:35:59.092744 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:59.092711    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b99845f6f-rqs6w"]
Oct 30 19:35:59.098000 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:59.097967    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b99845f6f-rqs6w"
Oct 30 19:35:59.112173 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:59.112115    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b99845f6f-rqs6w"]
Oct 30 19:35:59.184762 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:59.184727    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 19:35:59.184940 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:59.184854    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-stlgn\""
Oct 30 19:35:59.184940 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:59.184853    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 19:35:59.201511 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:59.201481    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2896aa38-c98c-4fa1-a8dd-3149d45e93b6-tmp\") pod \"kube-auth-proxy-5b99845f6f-rqs6w\" (UID: \"2896aa38-c98c-4fa1-a8dd-3149d45e93b6\") " pod="openshift-ingress/kube-auth-proxy-5b99845f6f-rqs6w"
Oct 30 19:35:59.201666 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:59.201526    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2896aa38-c98c-4fa1-a8dd-3149d45e93b6-tls-certs\") pod \"kube-auth-proxy-5b99845f6f-rqs6w\" (UID: \"2896aa38-c98c-4fa1-a8dd-3149d45e93b6\") " pod="openshift-ingress/kube-auth-proxy-5b99845f6f-rqs6w"
Oct 30 19:35:59.201666 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:59.201544    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n5cf\" (UniqueName: \"kubernetes.io/projected/2896aa38-c98c-4fa1-a8dd-3149d45e93b6-kube-api-access-7n5cf\") pod \"kube-auth-proxy-5b99845f6f-rqs6w\" (UID: \"2896aa38-c98c-4fa1-a8dd-3149d45e93b6\") " pod="openshift-ingress/kube-auth-proxy-5b99845f6f-rqs6w"
Oct 30 19:35:59.302593 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:59.302555    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2896aa38-c98c-4fa1-a8dd-3149d45e93b6-tmp\") pod \"kube-auth-proxy-5b99845f6f-rqs6w\" (UID: \"2896aa38-c98c-4fa1-a8dd-3149d45e93b6\") " pod="openshift-ingress/kube-auth-proxy-5b99845f6f-rqs6w"
Oct 30 19:35:59.302781 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:59.302619    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2896aa38-c98c-4fa1-a8dd-3149d45e93b6-tls-certs\") pod \"kube-auth-proxy-5b99845f6f-rqs6w\" (UID: \"2896aa38-c98c-4fa1-a8dd-3149d45e93b6\") " pod="openshift-ingress/kube-auth-proxy-5b99845f6f-rqs6w"
Oct 30 19:35:59.302781 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:59.302638    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n5cf\" (UniqueName: \"kubernetes.io/projected/2896aa38-c98c-4fa1-a8dd-3149d45e93b6-kube-api-access-7n5cf\") pod \"kube-auth-proxy-5b99845f6f-rqs6w\" (UID: \"2896aa38-c98c-4fa1-a8dd-3149d45e93b6\") " pod="openshift-ingress/kube-auth-proxy-5b99845f6f-rqs6w"
Oct 30 19:35:59.305070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:59.305021    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2896aa38-c98c-4fa1-a8dd-3149d45e93b6-tmp\") pod \"kube-auth-proxy-5b99845f6f-rqs6w\" (UID: \"2896aa38-c98c-4fa1-a8dd-3149d45e93b6\") " pod="openshift-ingress/kube-auth-proxy-5b99845f6f-rqs6w"
Oct 30 19:35:59.305284 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:59.305267    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2896aa38-c98c-4fa1-a8dd-3149d45e93b6-tls-certs\") pod \"kube-auth-proxy-5b99845f6f-rqs6w\" (UID: \"2896aa38-c98c-4fa1-a8dd-3149d45e93b6\") " pod="openshift-ingress/kube-auth-proxy-5b99845f6f-rqs6w"
Oct 30 19:35:59.326345 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:59.326319    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n5cf\" (UniqueName: \"kubernetes.io/projected/2896aa38-c98c-4fa1-a8dd-3149d45e93b6-kube-api-access-7n5cf\") pod \"kube-auth-proxy-5b99845f6f-rqs6w\" (UID: \"2896aa38-c98c-4fa1-a8dd-3149d45e93b6\") " pod="openshift-ingress/kube-auth-proxy-5b99845f6f-rqs6w"
Oct 30 19:35:59.410879 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:59.410849    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b99845f6f-rqs6w"
Oct 30 19:35:59.696338 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:35:59.696310    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b99845f6f-rqs6w"]
Oct 30 19:35:59.698675 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:35:59.698645    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2896aa38_c98c_4fa1_a8dd_3149d45e93b6.slice/crio-e7a6bd3abf01d49d48b65b72e9544d0c51a95ed4f93981cc8f28540c56904bac WatchSource:0}: Error finding container e7a6bd3abf01d49d48b65b72e9544d0c51a95ed4f93981cc8f28540c56904bac: Status 404 returned error can't find the container with id e7a6bd3abf01d49d48b65b72e9544d0c51a95ed4f93981cc8f28540c56904bac
Oct 30 19:36:00.062881 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:00.062800    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b99845f6f-rqs6w" event={"ID":"2896aa38-c98c-4fa1-a8dd-3149d45e93b6","Type":"ContainerStarted","Data":"e7a6bd3abf01d49d48b65b72e9544d0c51a95ed4f93981cc8f28540c56904bac"}
Oct 30 19:36:00.950597 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:00.950562    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mgr5d"]
Oct 30 19:36:00.955407 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:00.955382    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-mgr5d"
Oct 30 19:36:00.971207 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:00.971179    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gj5tc\""
Oct 30 19:36:01.079032 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:01.079002    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mgr5d"]
Oct 30 19:36:01.118541 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:01.118510    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94nmv\" (UniqueName: \"kubernetes.io/projected/cfbd0854-9f1e-412d-a98d-aa3806238fc6-kube-api-access-94nmv\") pod \"authorino-57f7f8659f-mgr5d\" (UID: \"cfbd0854-9f1e-412d-a98d-aa3806238fc6\") " pod="kuadrant-system/authorino-57f7f8659f-mgr5d"
Oct 30 19:36:01.219579 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:01.219501    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94nmv\" (UniqueName: \"kubernetes.io/projected/cfbd0854-9f1e-412d-a98d-aa3806238fc6-kube-api-access-94nmv\") pod \"authorino-57f7f8659f-mgr5d\" (UID: \"cfbd0854-9f1e-412d-a98d-aa3806238fc6\") " pod="kuadrant-system/authorino-57f7f8659f-mgr5d"
Oct 30 19:36:01.241745 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:01.241716    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94nmv\" (UniqueName: \"kubernetes.io/projected/cfbd0854-9f1e-412d-a98d-aa3806238fc6-kube-api-access-94nmv\") pod \"authorino-57f7f8659f-mgr5d\" (UID: \"cfbd0854-9f1e-412d-a98d-aa3806238fc6\") " pod="kuadrant-system/authorino-57f7f8659f-mgr5d"
Oct 30 19:36:01.283524 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:01.283495    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gj5tc\""
Oct 30 19:36:01.290597 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:01.290572    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-mgr5d"
Oct 30 19:36:01.589548 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:01.589481    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mgr5d"]
Oct 30 19:36:01.593172 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:36:01.593148    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfbd0854_9f1e_412d_a98d_aa3806238fc6.slice/crio-b2ee84845f2beb09e767946b5b6aed7f9ad7f5bd82d874a62becdf9d62b48c8a WatchSource:0}: Error finding container b2ee84845f2beb09e767946b5b6aed7f9ad7f5bd82d874a62becdf9d62b48c8a: Status 404 returned error can't find the container with id b2ee84845f2beb09e767946b5b6aed7f9ad7f5bd82d874a62becdf9d62b48c8a
Oct 30 19:36:02.032059 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:02.032013    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mgclc"
Oct 30 19:36:02.051885 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:02.051865    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 19:36:02.073996 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:02.073952    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-mgr5d" event={"ID":"cfbd0854-9f1e-412d-a98d-aa3806238fc6","Type":"ContainerStarted","Data":"b2ee84845f2beb09e767946b5b6aed7f9ad7f5bd82d874a62becdf9d62b48c8a"}
Oct 30 19:36:03.080720 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:03.080688    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b99845f6f-rqs6w" event={"ID":"2896aa38-c98c-4fa1-a8dd-3149d45e93b6","Type":"ContainerStarted","Data":"dbdf41e879ef4f19b052da8ee4670da373cdb7165232bb5f0d8026044595d232"}
Oct 30 19:36:04.040390 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:04.040359    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6pvgl"
Oct 30 19:36:04.068435 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:04.068389    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b99845f6f-rqs6w" podStartSLOduration=2.720752562 podStartE2EDuration="5.068374286s" totalImagesPullingTime="2.347621724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:03.114001185 +0000 UTC m=+242.029608042" watchObservedRunningTime="2026-10-30 19:36:04.068374286 +0000 UTC m=+242.983981141"
Oct 30 19:36:04.086284 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:04.086236    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-mgr5d" event={"ID":"cfbd0854-9f1e-412d-a98d-aa3806238fc6","Type":"ContainerStarted","Data":"5900295413de1edaddf77698d8ec53f84a415d9c7a65b575c6a64dce472aaa47"}
Oct 30 19:36:04.113805 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:04.113753    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-mgr5d" podStartSLOduration=2.159472153 podStartE2EDuration="4.113738046s" totalImagesPullingTime="1.954265893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:04.11300017 +0000 UTC m=+243.028607027" watchObservedRunningTime="2026-10-30 19:36:04.113738046 +0000 UTC m=+243.029344901"
Oct 30 19:36:04.287752 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:36:04.287715    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:36:05.192960 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:05.192929    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mgr5d"]
Oct 30 19:36:06.094011 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:06.093974    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-mgr5d" podUID="cfbd0854-9f1e-412d-a98d-aa3806238fc6" containerName="authorino" containerID="cri-o://5900295413de1edaddf77698d8ec53f84a415d9c7a65b575c6a64dce472aaa47" gracePeriod=30
Oct 30 19:36:06.338247 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:06.338209    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-mgr5d"
Oct 30 19:36:06.357447 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:06.357379    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfbd0854-9f1e-412d-a98d-aa3806238fc6-kube-api-access-94nmv\" (UniqueName: \"kubernetes.io/projected/cfbd0854-9f1e-412d-a98d-aa3806238fc6-kube-api-access-94nmv\") pod \"cfbd0854-9f1e-412d-a98d-aa3806238fc6\" (UID: \"cfbd0854-9f1e-412d-a98d-aa3806238fc6\") "
Oct 30 19:36:06.359616 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:06.359588    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfbd0854-9f1e-412d-a98d-aa3806238fc6-kube-api-access-94nmv" pod "cfbd0854-9f1e-412d-a98d-aa3806238fc6" (UID: "cfbd0854-9f1e-412d-a98d-aa3806238fc6"). InnerVolumeSpecName "kube-api-access-94nmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:06.458782 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:06.458745    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94nmv\" (UniqueName: \"kubernetes.io/projected/cfbd0854-9f1e-412d-a98d-aa3806238fc6-kube-api-access-94nmv\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:07.098705 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:07.098665    6030 generic.go:356] "Generic (PLEG): container finished" podID="cfbd0854-9f1e-412d-a98d-aa3806238fc6" containerID="5900295413de1edaddf77698d8ec53f84a415d9c7a65b575c6a64dce472aaa47" exitCode=0
Oct 30 19:36:07.098882 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:07.098710    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-mgr5d"
Oct 30 19:36:07.098882 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:07.098716    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-mgr5d" event={"ID":"cfbd0854-9f1e-412d-a98d-aa3806238fc6","Type":"ContainerDied","Data":"5900295413de1edaddf77698d8ec53f84a415d9c7a65b575c6a64dce472aaa47"}
Oct 30 19:36:07.098882 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:07.098762    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-mgr5d" event={"ID":"cfbd0854-9f1e-412d-a98d-aa3806238fc6","Type":"ContainerDied","Data":"b2ee84845f2beb09e767946b5b6aed7f9ad7f5bd82d874a62becdf9d62b48c8a"}
Oct 30 19:36:07.098882 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:07.098784    6030 scope.go:122] "RemoveContainer" containerID="5900295413de1edaddf77698d8ec53f84a415d9c7a65b575c6a64dce472aaa47"
Oct 30 19:36:07.107380 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:07.107362    6030 scope.go:122] "RemoveContainer" containerID="5900295413de1edaddf77698d8ec53f84a415d9c7a65b575c6a64dce472aaa47"
Oct 30 19:36:07.107658 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:36:07.107635    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5900295413de1edaddf77698d8ec53f84a415d9c7a65b575c6a64dce472aaa47\": container with ID starting with 5900295413de1edaddf77698d8ec53f84a415d9c7a65b575c6a64dce472aaa47 not found: ID does not exist" containerID="5900295413de1edaddf77698d8ec53f84a415d9c7a65b575c6a64dce472aaa47"
Oct 30 19:36:07.107747 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:07.107670    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5900295413de1edaddf77698d8ec53f84a415d9c7a65b575c6a64dce472aaa47"} err="failed to get container status \"5900295413de1edaddf77698d8ec53f84a415d9c7a65b575c6a64dce472aaa47\": rpc error: code = NotFound desc = could not find container \"5900295413de1edaddf77698d8ec53f84a415d9c7a65b575c6a64dce472aaa47\": container with ID starting with 5900295413de1edaddf77698d8ec53f84a415d9c7a65b575c6a64dce472aaa47 not found: ID does not exist"
Oct 30 19:36:07.172239 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:07.172200    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mgr5d"]
Oct 30 19:36:07.178424 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:07.178399    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mgr5d"]
Oct 30 19:36:07.198791 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:07.198761    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfbd0854-9f1e-412d-a98d-aa3806238fc6" path="/var/lib/kubelet/pods/cfbd0854-9f1e-412d-a98d-aa3806238fc6/volumes"
Oct 30 19:36:34.286949 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:36:34.286911    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:36:42.585256 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.585181    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cf964748-d8zhr"]
Oct 30 19:36:42.585612 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.585504    6030 cpu_manager.go:414] "containerMap: removing container" podUID="cfbd0854-9f1e-412d-a98d-aa3806238fc6" containerName="authorino"
Oct 30 19:36:42.585612 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.585515    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfbd0854-9f1e-412d-a98d-aa3806238fc6" containerName="authorino"
Oct 30 19:36:42.585612 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.585568    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfbd0854-9f1e-412d-a98d-aa3806238fc6" containerName="authorino"
Oct 30 19:36:42.588224 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.588203    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cf964748-d8zhr"
Oct 30 19:36:42.593804 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.593785    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 19:36:42.593804 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.593785    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 19:36:42.594701 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.594681    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 19:36:42.594809 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.594755    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 19:36:42.594809 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.594792    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2w2d4\""
Oct 30 19:36:42.594993 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.594975    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 19:36:42.601814 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.601791    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cf964748-d8zhr"]
Oct 30 19:36:42.648907 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.648873    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89wwf\" (UniqueName: \"kubernetes.io/projected/c2b7322b-ec51-4436-bedb-b4a923ffdf75-kube-api-access-89wwf\") pod \"maas-controller-5cf964748-d8zhr\" (UID: \"c2b7322b-ec51-4436-bedb-b4a923ffdf75\") " pod="opendatahub/maas-controller-5cf964748-d8zhr"
Oct 30 19:36:42.648907 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.648912    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b7322b-ec51-4436-bedb-b4a923ffdf75-metrics-certs\") pod \"maas-controller-5cf964748-d8zhr\" (UID: \"c2b7322b-ec51-4436-bedb-b4a923ffdf75\") " pod="opendatahub/maas-controller-5cf964748-d8zhr"
Oct 30 19:36:42.649162 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.648942    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2b7322b-ec51-4436-bedb-b4a923ffdf75-webhook-cert\") pod \"maas-controller-5cf964748-d8zhr\" (UID: \"c2b7322b-ec51-4436-bedb-b4a923ffdf75\") " pod="opendatahub/maas-controller-5cf964748-d8zhr"
Oct 30 19:36:42.750255 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.750218    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89wwf\" (UniqueName: \"kubernetes.io/projected/c2b7322b-ec51-4436-bedb-b4a923ffdf75-kube-api-access-89wwf\") pod \"maas-controller-5cf964748-d8zhr\" (UID: \"c2b7322b-ec51-4436-bedb-b4a923ffdf75\") " pod="opendatahub/maas-controller-5cf964748-d8zhr"
Oct 30 19:36:42.750452 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.750260    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b7322b-ec51-4436-bedb-b4a923ffdf75-metrics-certs\") pod \"maas-controller-5cf964748-d8zhr\" (UID: \"c2b7322b-ec51-4436-bedb-b4a923ffdf75\") " pod="opendatahub/maas-controller-5cf964748-d8zhr"
Oct 30 19:36:42.750452 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.750303    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2b7322b-ec51-4436-bedb-b4a923ffdf75-webhook-cert\") pod \"maas-controller-5cf964748-d8zhr\" (UID: \"c2b7322b-ec51-4436-bedb-b4a923ffdf75\") " pod="opendatahub/maas-controller-5cf964748-d8zhr"
Oct 30 19:36:42.752893 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.752869    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2b7322b-ec51-4436-bedb-b4a923ffdf75-webhook-cert\") pod \"maas-controller-5cf964748-d8zhr\" (UID: \"c2b7322b-ec51-4436-bedb-b4a923ffdf75\") " pod="opendatahub/maas-controller-5cf964748-d8zhr"
Oct 30 19:36:42.753039 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.753017    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b7322b-ec51-4436-bedb-b4a923ffdf75-metrics-certs\") pod \"maas-controller-5cf964748-d8zhr\" (UID: \"c2b7322b-ec51-4436-bedb-b4a923ffdf75\") " pod="opendatahub/maas-controller-5cf964748-d8zhr"
Oct 30 19:36:42.764030 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.764004    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89wwf\" (UniqueName: \"kubernetes.io/projected/c2b7322b-ec51-4436-bedb-b4a923ffdf75-kube-api-access-89wwf\") pod \"maas-controller-5cf964748-d8zhr\" (UID: \"c2b7322b-ec51-4436-bedb-b4a923ffdf75\") " pod="opendatahub/maas-controller-5cf964748-d8zhr"
Oct 30 19:36:42.900174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:42.900142    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cf964748-d8zhr"
Oct 30 19:36:43.174127 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:43.174055    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cf964748-d8zhr"]
Oct 30 19:36:43.177099 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:36:43.177074    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b7322b_ec51_4436_bedb_b4a923ffdf75.slice/crio-2e19beaa2dbdc466706cb970bf722686fc80937f652849388b49223b2255b4a5 WatchSource:0}: Error finding container 2e19beaa2dbdc466706cb970bf722686fc80937f652849388b49223b2255b4a5: Status 404 returned error can't find the container with id 2e19beaa2dbdc466706cb970bf722686fc80937f652849388b49223b2255b4a5
Oct 30 19:36:43.243641 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:43.243605    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cf964748-d8zhr" event={"ID":"c2b7322b-ec51-4436-bedb-b4a923ffdf75","Type":"ContainerStarted","Data":"2e19beaa2dbdc466706cb970bf722686fc80937f652849388b49223b2255b4a5"}
Oct 30 19:36:48.266410 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:48.266370    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cf964748-d8zhr" event={"ID":"c2b7322b-ec51-4436-bedb-b4a923ffdf75","Type":"ContainerStarted","Data":"e3d30d41bb132f6ecc7e468cce8983fae45fa2b10f099341f11309ccee75f834"}
Oct 30 19:36:48.266844 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:48.266469    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cf964748-d8zhr"
Oct 30 19:36:48.292382 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:48.292325    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5cf964748-d8zhr" podStartSLOduration=2.048363297 podStartE2EDuration="6.29231025s" totalImagesPullingTime="4.243946953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:48.29162654 +0000 UTC m=+287.207233397" watchObservedRunningTime="2026-10-30 19:36:48.29231025 +0000 UTC m=+287.207917107"
Oct 30 19:36:48.862210 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:48.862179    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"]
Oct 30 19:36:48.865691 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:48.865667    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"
Oct 30 19:36:48.882633 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:48.882600    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 19:36:48.882819 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:48.882748    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 19:36:48.883898 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:48.883872    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"]
Oct 30 19:36:48.925991 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:48.925965    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 19:36:48.926190 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:48.926107    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-j8cpn\""
Oct 30 19:36:49.004573 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:49.004538    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9b909a-d885-4ff2-8032-8079ac535f5a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-89xm5\" (UID: \"6a9b909a-d885-4ff2-8032-8079ac535f5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"
Oct 30 19:36:49.004573 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:49.004578    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9p2l\" (UniqueName: \"kubernetes.io/projected/6a9b909a-d885-4ff2-8032-8079ac535f5a-kube-api-access-p9p2l\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-89xm5\" (UID: \"6a9b909a-d885-4ff2-8032-8079ac535f5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"
Oct 30 19:36:49.004802 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:49.004603    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a9b909a-d885-4ff2-8032-8079ac535f5a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-89xm5\" (UID: \"6a9b909a-d885-4ff2-8032-8079ac535f5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"
Oct 30 19:36:49.004802 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:49.004720    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6a9b909a-d885-4ff2-8032-8079ac535f5a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-89xm5\" (UID: \"6a9b909a-d885-4ff2-8032-8079ac535f5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"
Oct 30 19:36:49.105968 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:49.105933    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a9b909a-d885-4ff2-8032-8079ac535f5a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-89xm5\" (UID: \"6a9b909a-d885-4ff2-8032-8079ac535f5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"
Oct 30 19:36:49.106161 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:49.105977    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6a9b909a-d885-4ff2-8032-8079ac535f5a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-89xm5\" (UID: \"6a9b909a-d885-4ff2-8032-8079ac535f5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"
Oct 30 19:36:49.106161 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:49.106021    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9b909a-d885-4ff2-8032-8079ac535f5a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-89xm5\" (UID: \"6a9b909a-d885-4ff2-8032-8079ac535f5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"
Oct 30 19:36:49.106161 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:49.106055    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9p2l\" (UniqueName: \"kubernetes.io/projected/6a9b909a-d885-4ff2-8032-8079ac535f5a-kube-api-access-p9p2l\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-89xm5\" (UID: \"6a9b909a-d885-4ff2-8032-8079ac535f5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"
Oct 30 19:36:49.106941 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:49.106919    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9b909a-d885-4ff2-8032-8079ac535f5a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-89xm5\" (UID: \"6a9b909a-d885-4ff2-8032-8079ac535f5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"
Oct 30 19:36:49.107183 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:49.107165    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6a9b909a-d885-4ff2-8032-8079ac535f5a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-89xm5\" (UID: \"6a9b909a-d885-4ff2-8032-8079ac535f5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"
Oct 30 19:36:49.108540 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:49.108516    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a9b909a-d885-4ff2-8032-8079ac535f5a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-89xm5\" (UID: \"6a9b909a-d885-4ff2-8032-8079ac535f5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"
Oct 30 19:36:49.120855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:49.120796    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9p2l\" (UniqueName: \"kubernetes.io/projected/6a9b909a-d885-4ff2-8032-8079ac535f5a-kube-api-access-p9p2l\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-89xm5\" (UID: \"6a9b909a-d885-4ff2-8032-8079ac535f5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"
Oct 30 19:36:49.179558 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:49.179507    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"
Oct 30 19:36:49.456010 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:49.455976    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"]
Oct 30 19:36:49.458642 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:36:49.458616    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9b909a_d885_4ff2_8032_8079ac535f5a.slice/crio-487d0aaa43ef81cab5f6317675824e803209e418ddb5e5127a1a8f968a527c93 WatchSource:0}: Error finding container 487d0aaa43ef81cab5f6317675824e803209e418ddb5e5127a1a8f968a527c93: Status 404 returned error can't find the container with id 487d0aaa43ef81cab5f6317675824e803209e418ddb5e5127a1a8f968a527c93
Oct 30 19:36:50.274441 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:50.274409    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5" event={"ID":"6a9b909a-d885-4ff2-8032-8079ac535f5a","Type":"ContainerStarted","Data":"487d0aaa43ef81cab5f6317675824e803209e418ddb5e5127a1a8f968a527c93"}
Oct 30 19:36:55.858071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:55.856758    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-grxfc"]
Oct 30 19:36:55.882990 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:55.882945    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-grxfc"]
Oct 30 19:36:55.883185 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:55.883113    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-grxfc"
Oct 30 19:36:55.895036 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:55.895012    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 19:36:55.895036 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:55.895026    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-jcld9\""
Oct 30 19:36:55.977675 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:55.977642    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39be0ba6-63fe-4329-bfbd-7eed3d36be55-cert\") pod \"kserve-controller-manager-776cc66d47-grxfc\" (UID: \"39be0ba6-63fe-4329-bfbd-7eed3d36be55\") " pod="opendatahub/kserve-controller-manager-776cc66d47-grxfc"
Oct 30 19:36:55.977867 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:55.977693    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grl6s\" (UniqueName: \"kubernetes.io/projected/39be0ba6-63fe-4329-bfbd-7eed3d36be55-kube-api-access-grl6s\") pod \"kserve-controller-manager-776cc66d47-grxfc\" (UID: \"39be0ba6-63fe-4329-bfbd-7eed3d36be55\") " pod="opendatahub/kserve-controller-manager-776cc66d47-grxfc"
Oct 30 19:36:56.078920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.078885    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39be0ba6-63fe-4329-bfbd-7eed3d36be55-cert\") pod \"kserve-controller-manager-776cc66d47-grxfc\" (UID: \"39be0ba6-63fe-4329-bfbd-7eed3d36be55\") " pod="opendatahub/kserve-controller-manager-776cc66d47-grxfc"
Oct 30 19:36:56.078920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.078925    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grl6s\" (UniqueName: \"kubernetes.io/projected/39be0ba6-63fe-4329-bfbd-7eed3d36be55-kube-api-access-grl6s\") pod \"kserve-controller-manager-776cc66d47-grxfc\" (UID: \"39be0ba6-63fe-4329-bfbd-7eed3d36be55\") " pod="opendatahub/kserve-controller-manager-776cc66d47-grxfc"
Oct 30 19:36:56.082232 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.082203    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39be0ba6-63fe-4329-bfbd-7eed3d36be55-cert\") pod \"kserve-controller-manager-776cc66d47-grxfc\" (UID: \"39be0ba6-63fe-4329-bfbd-7eed3d36be55\") " pod="opendatahub/kserve-controller-manager-776cc66d47-grxfc"
Oct 30 19:36:56.094674 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.094644    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grl6s\" (UniqueName: \"kubernetes.io/projected/39be0ba6-63fe-4329-bfbd-7eed3d36be55-kube-api-access-grl6s\") pod \"kserve-controller-manager-776cc66d47-grxfc\" (UID: \"39be0ba6-63fe-4329-bfbd-7eed3d36be55\") " pod="opendatahub/kserve-controller-manager-776cc66d47-grxfc"
Oct 30 19:36:56.200946 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.200899    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-grxfc"
Oct 30 19:36:56.517705 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.516679    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-grxfc"]
Oct 30 19:36:56.521209 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:36:56.521173    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39be0ba6_63fe_4329_bfbd_7eed3d36be55.slice/crio-f18c7b183a868e9392e7bba073123752301d19c8ed7802ca35080cb846d8c611 WatchSource:0}: Error finding container f18c7b183a868e9392e7bba073123752301d19c8ed7802ca35080cb846d8c611: Status 404 returned error can't find the container with id f18c7b183a868e9392e7bba073123752301d19c8ed7802ca35080cb846d8c611
Oct 30 19:36:56.603254 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.603129    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"]
Oct 30 19:36:56.608766 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.608460    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.643095 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.639709    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"]
Oct 30 19:36:56.784510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.784433    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/618c01b7-db1f-47a4-be57-aa701de1a286-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.784678 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.784505    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc6d9\" (UniqueName: \"kubernetes.io/projected/618c01b7-db1f-47a4-be57-aa701de1a286-kube-api-access-wc6d9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.784678 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.784546    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/618c01b7-db1f-47a4-be57-aa701de1a286-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.784678 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.784572    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618c01b7-db1f-47a4-be57-aa701de1a286-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.784678 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.784598    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.784678 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.784613    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.784678 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.784652    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.885289 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.885246    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/618c01b7-db1f-47a4-be57-aa701de1a286-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.885683 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.885314    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc6d9\" (UniqueName: \"kubernetes.io/projected/618c01b7-db1f-47a4-be57-aa701de1a286-kube-api-access-wc6d9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.885683 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.885353    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/618c01b7-db1f-47a4-be57-aa701de1a286-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.885683 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.885382    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618c01b7-db1f-47a4-be57-aa701de1a286-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.885683 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.885417    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.885683 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.885431    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.885683 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.885468    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.886598 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.886559    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/618c01b7-db1f-47a4-be57-aa701de1a286-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.888638 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.888609    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.889337 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.889309    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.889446 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.889394    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.890541 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.890519    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/618c01b7-db1f-47a4-be57-aa701de1a286-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.900413 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.900382    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618c01b7-db1f-47a4-be57-aa701de1a286-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.900973 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.900941    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc6d9\" (UniqueName: \"kubernetes.io/projected/618c01b7-db1f-47a4-be57-aa701de1a286-kube-api-access-wc6d9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p8jcd\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:56.924621 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:56.924572    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:57.225140 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:57.224415    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"]
Oct 30 19:36:57.227995 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:36:57.227133    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618c01b7_db1f_47a4_be57_aa701de1a286.slice/crio-190412ce73d2effadbbf3e66134958d87df3811b2f2739413407ad119d5ce315 WatchSource:0}: Error finding container 190412ce73d2effadbbf3e66134958d87df3811b2f2739413407ad119d5ce315: Status 404 returned error can't find the container with id 190412ce73d2effadbbf3e66134958d87df3811b2f2739413407ad119d5ce315
Oct 30 19:36:57.229749 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:57.229710    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:36:57.229848 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:57.229776    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:36:57.314658 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:57.314622    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd" event={"ID":"618c01b7-db1f-47a4-be57-aa701de1a286","Type":"ContainerStarted","Data":"48520e40bd56963a5442ca72c5064573016b08a05765f1ed0edc631b96431407"}
Oct 30 19:36:57.314818 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:57.314684    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd" event={"ID":"618c01b7-db1f-47a4-be57-aa701de1a286","Type":"ContainerStarted","Data":"190412ce73d2effadbbf3e66134958d87df3811b2f2739413407ad119d5ce315"}
Oct 30 19:36:57.314818 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:57.314745    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:57.317330 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:57.317291    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-grxfc" event={"ID":"39be0ba6-63fe-4329-bfbd-7eed3d36be55","Type":"ContainerStarted","Data":"f18c7b183a868e9392e7bba073123752301d19c8ed7802ca35080cb846d8c611"}
Oct 30 19:36:57.347007 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:57.346939    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd" podStartSLOduration=1.346919695 podStartE2EDuration="1.346919695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:57.346463016 +0000 UTC m=+296.262069875" watchObservedRunningTime="2026-10-30 19:36:57.346919695 +0000 UTC m=+296.262526553"
Oct 30 19:36:58.325200 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.325156    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 19:36:58.391668 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.391631    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"]
Oct 30 19:36:58.391942 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.391887    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk" podUID="ceaabe36-9009-4d0a-9a0a-c48b23bfd93b" containerName="discovery" containerID="cri-o://16f3b83751bef482f5b34ab95c562a9cb93147ba2cf79c1e1941b90ebd4608d3" gracePeriod=30
Oct 30 19:36:58.674368 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.674342    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:36:58.805287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.804582    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-csr-ca-configmap\") pod \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") "
Oct 30 19:36:58.805287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.804650    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-kube-api-access-xtwrh\" (UniqueName: \"kubernetes.io/projected/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-kube-api-access-xtwrh\") pod \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") "
Oct 30 19:36:58.805287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.804689    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-kubeconfig\") pod \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") "
Oct 30 19:36:58.805287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.804720    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-csr-dns-cert\") pod \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") "
Oct 30 19:36:58.805287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.804752    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-cacerts\" (UniqueName: \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-cacerts\") pod \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") "
Oct 30 19:36:58.805287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.804780    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-local-certs\") pod \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") "
Oct 30 19:36:58.805287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.804815    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-token\" (UniqueName: \"kubernetes.io/projected/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-token\") pod \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\" (UID: \"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b\") "
Oct 30 19:36:58.805866 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.805571    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-csr-ca-configmap" pod "ceaabe36-9009-4d0a-9a0a-c48b23bfd93b" (UID: "ceaabe36-9009-4d0a-9a0a-c48b23bfd93b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:58.809216 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.809173    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-local-certs" pod "ceaabe36-9009-4d0a-9a0a-c48b23bfd93b" (UID: "ceaabe36-9009-4d0a-9a0a-c48b23bfd93b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:58.809339 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.809205    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-cacerts" pod "ceaabe36-9009-4d0a-9a0a-c48b23bfd93b" (UID: "ceaabe36-9009-4d0a-9a0a-c48b23bfd93b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:58.811104 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.810939    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-kubeconfig" pod "ceaabe36-9009-4d0a-9a0a-c48b23bfd93b" (UID: "ceaabe36-9009-4d0a-9a0a-c48b23bfd93b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:58.811104 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.810987    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-kube-api-access-xtwrh" pod "ceaabe36-9009-4d0a-9a0a-c48b23bfd93b" (UID: "ceaabe36-9009-4d0a-9a0a-c48b23bfd93b"). InnerVolumeSpecName "kube-api-access-xtwrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:58.811104 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.811077    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-csr-dns-cert" pod "ceaabe36-9009-4d0a-9a0a-c48b23bfd93b" (UID: "ceaabe36-9009-4d0a-9a0a-c48b23bfd93b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:58.811601 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.811575    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-token" pod "ceaabe36-9009-4d0a-9a0a-c48b23bfd93b" (UID: "ceaabe36-9009-4d0a-9a0a-c48b23bfd93b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:58.906518 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.906478    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtwrh\" (UniqueName: \"kubernetes.io/projected/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-kube-api-access-xtwrh\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:58.906518 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.906518    6030 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-kubeconfig\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:58.906776 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.906535    6030 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-csr-dns-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:58.906776 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.906550    6030 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-cacerts\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:58.906776 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.906564    6030 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-local-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:58.906776 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.906577    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:58.906776 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:58.906590    6030 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b-istio-csr-ca-configmap\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:59.272158 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:59.272127    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cf964748-d8zhr"
Oct 30 19:36:59.330178 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:59.330138    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-grxfc" event={"ID":"39be0ba6-63fe-4329-bfbd-7eed3d36be55","Type":"ContainerStarted","Data":"1c822a2dbc8d2fde75a8066af65d6a62796ff850ed8845ca9b1161871320299b"}
Oct 30 19:36:59.330639 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:59.330213    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-grxfc"
Oct 30 19:36:59.332476 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:59.332444    6030 generic.go:356] "Generic (PLEG): container finished" podID="ceaabe36-9009-4d0a-9a0a-c48b23bfd93b" containerID="16f3b83751bef482f5b34ab95c562a9cb93147ba2cf79c1e1941b90ebd4608d3" exitCode=0
Oct 30 19:36:59.332616 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:59.332487    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk" event={"ID":"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b","Type":"ContainerDied","Data":"16f3b83751bef482f5b34ab95c562a9cb93147ba2cf79c1e1941b90ebd4608d3"}
Oct 30 19:36:59.332616 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:59.332502    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"
Oct 30 19:36:59.332616 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:59.332542    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk" event={"ID":"ceaabe36-9009-4d0a-9a0a-c48b23bfd93b","Type":"ContainerDied","Data":"72ff1b6566d9b7e2f872a74c18fb7add20062f2b8d9635da338561571aeceb9a"}
Oct 30 19:36:59.332616 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:59.332568    6030 scope.go:122] "RemoveContainer" containerID="16f3b83751bef482f5b34ab95c562a9cb93147ba2cf79c1e1941b90ebd4608d3"
Oct 30 19:36:59.343420 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:59.343399    6030 scope.go:122] "RemoveContainer" containerID="16f3b83751bef482f5b34ab95c562a9cb93147ba2cf79c1e1941b90ebd4608d3"
Oct 30 19:36:59.343838 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:36:59.343761    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f3b83751bef482f5b34ab95c562a9cb93147ba2cf79c1e1941b90ebd4608d3\": container with ID starting with 16f3b83751bef482f5b34ab95c562a9cb93147ba2cf79c1e1941b90ebd4608d3 not found: ID does not exist" containerID="16f3b83751bef482f5b34ab95c562a9cb93147ba2cf79c1e1941b90ebd4608d3"
Oct 30 19:36:59.343838 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:59.343806    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f3b83751bef482f5b34ab95c562a9cb93147ba2cf79c1e1941b90ebd4608d3"} err="failed to get container status \"16f3b83751bef482f5b34ab95c562a9cb93147ba2cf79c1e1941b90ebd4608d3\": rpc error: code = NotFound desc = could not find container \"16f3b83751bef482f5b34ab95c562a9cb93147ba2cf79c1e1941b90ebd4608d3\": container with ID starting with 16f3b83751bef482f5b34ab95c562a9cb93147ba2cf79c1e1941b90ebd4608d3 not found: ID does not exist"
Oct 30 19:36:59.388782 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:59.388718    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-grxfc" podStartSLOduration=2.168408815 podStartE2EDuration="4.388700685s" totalImagesPullingTime="2.22029187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:59.38833674 +0000 UTC m=+298.303943597" watchObservedRunningTime="2026-10-30 19:36:59.388700685 +0000 UTC m=+298.304307541"
Oct 30 19:36:59.417662 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:59.417624    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"]
Oct 30 19:36:59.426068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:36:59.426009    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-s74hk"]
Oct 30 19:37:01.200435 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:01.200403    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceaabe36-9009-4d0a-9a0a-c48b23bfd93b" path="/var/lib/kubelet/pods/ceaabe36-9009-4d0a-9a0a-c48b23bfd93b/volumes"
Oct 30 19:37:03.352903 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:03.352865    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5" event={"ID":"6a9b909a-d885-4ff2-8032-8079ac535f5a","Type":"ContainerStarted","Data":"ccb8cf83ba6dfaf825599a937419e105f0dba0cd3a4b592c47aa1bbffe693216"}
Oct 30 19:37:03.353338 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:03.353064    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"
Oct 30 19:37:03.378111 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:03.378038    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5" podStartSLOduration=2.252931481 podStartE2EDuration="15.378021974s" totalImagesPullingTime="13.125090493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:03.3772795 +0000 UTC m=+302.292886356" watchObservedRunningTime="2026-10-30 19:37:03.378021974 +0000 UTC m=+302.293628808"
Oct 30 19:37:04.287743 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:37:04.287698    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:37:14.363053 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:14.363023    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-89xm5"
Oct 30 19:37:20.437027 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.436993    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"]
Oct 30 19:37:20.437515 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.437362    6030 cpu_manager.go:414] "containerMap: removing container" podUID="ceaabe36-9009-4d0a-9a0a-c48b23bfd93b" containerName="discovery"
Oct 30 19:37:20.437515 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.437374    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceaabe36-9009-4d0a-9a0a-c48b23bfd93b" containerName="discovery"
Oct 30 19:37:20.437515 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.437446    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceaabe36-9009-4d0a-9a0a-c48b23bfd93b" containerName="discovery"
Oct 30 19:37:20.555230 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.555195    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"]
Oct 30 19:37:20.555385 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.555316    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:20.561708 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.561682    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:37:20.561904 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.561780    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 19:37:20.561904 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.561847    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d892r\""
Oct 30 19:37:20.562063 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.562023    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:37:20.562437 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.562167    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:37:20.572610 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.572586    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/826c832f-1687-4592-a7d8-a4ce50c48c78-maas-api-tls\") pod \"maas-api-7cb9b5d67c-phcbd\" (UID: \"826c832f-1687-4592-a7d8-a4ce50c48c78\") " pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:20.572718 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.572624    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/826c832f-1687-4592-a7d8-a4ce50c48c78-metrics-certs\") pod \"maas-api-7cb9b5d67c-phcbd\" (UID: \"826c832f-1687-4592-a7d8-a4ce50c48c78\") " pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:20.572718 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.572655    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6khm\" (UniqueName: \"kubernetes.io/projected/826c832f-1687-4592-a7d8-a4ce50c48c78-kube-api-access-t6khm\") pod \"maas-api-7cb9b5d67c-phcbd\" (UID: \"826c832f-1687-4592-a7d8-a4ce50c48c78\") " pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:20.572800 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.572772    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/826c832f-1687-4592-a7d8-a4ce50c48c78-openshift-service-ca\") pod \"maas-api-7cb9b5d67c-phcbd\" (UID: \"826c832f-1687-4592-a7d8-a4ce50c48c78\") " pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:20.673348 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.673317    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/826c832f-1687-4592-a7d8-a4ce50c48c78-metrics-certs\") pod \"maas-api-7cb9b5d67c-phcbd\" (UID: \"826c832f-1687-4592-a7d8-a4ce50c48c78\") " pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:20.673550 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.673379    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6khm\" (UniqueName: \"kubernetes.io/projected/826c832f-1687-4592-a7d8-a4ce50c48c78-kube-api-access-t6khm\") pod \"maas-api-7cb9b5d67c-phcbd\" (UID: \"826c832f-1687-4592-a7d8-a4ce50c48c78\") " pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:20.673550 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.673449    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/826c832f-1687-4592-a7d8-a4ce50c48c78-openshift-service-ca\") pod \"maas-api-7cb9b5d67c-phcbd\" (UID: \"826c832f-1687-4592-a7d8-a4ce50c48c78\") " pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:20.673550 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.673472    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/826c832f-1687-4592-a7d8-a4ce50c48c78-maas-api-tls\") pod \"maas-api-7cb9b5d67c-phcbd\" (UID: \"826c832f-1687-4592-a7d8-a4ce50c48c78\") " pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:20.674163 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.674141    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/826c832f-1687-4592-a7d8-a4ce50c48c78-openshift-service-ca\") pod \"maas-api-7cb9b5d67c-phcbd\" (UID: \"826c832f-1687-4592-a7d8-a4ce50c48c78\") " pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:20.675982 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.675957    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/826c832f-1687-4592-a7d8-a4ce50c48c78-maas-api-tls\") pod \"maas-api-7cb9b5d67c-phcbd\" (UID: \"826c832f-1687-4592-a7d8-a4ce50c48c78\") " pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:20.676134 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.676112    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/826c832f-1687-4592-a7d8-a4ce50c48c78-metrics-certs\") pod \"maas-api-7cb9b5d67c-phcbd\" (UID: \"826c832f-1687-4592-a7d8-a4ce50c48c78\") " pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:20.686414 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.686391    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6khm\" (UniqueName: \"kubernetes.io/projected/826c832f-1687-4592-a7d8-a4ce50c48c78-kube-api-access-t6khm\") pod \"maas-api-7cb9b5d67c-phcbd\" (UID: \"826c832f-1687-4592-a7d8-a4ce50c48c78\") " pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:20.889655 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:20.889616    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:21.214859 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:21.214788    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"]
Oct 30 19:37:21.217648 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:37:21.217614    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826c832f_1687_4592_a7d8_a4ce50c48c78.slice/crio-07099d346f4fa471a485feb1af0c62faaacaae21eb2eded5d4af0ec049da6cd8 WatchSource:0}: Error finding container 07099d346f4fa471a485feb1af0c62faaacaae21eb2eded5d4af0ec049da6cd8: Status 404 returned error can't find the container with id 07099d346f4fa471a485feb1af0c62faaacaae21eb2eded5d4af0ec049da6cd8
Oct 30 19:37:21.424881 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:21.424852    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd" event={"ID":"826c832f-1687-4592-a7d8-a4ce50c48c78","Type":"ContainerStarted","Data":"07099d346f4fa471a485feb1af0c62faaacaae21eb2eded5d4af0ec049da6cd8"}
Oct 30 19:37:26.446212 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:26.446174    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd" event={"ID":"826c832f-1687-4592-a7d8-a4ce50c48c78","Type":"ContainerStarted","Data":"aa7f650666602f2f6eaa52ad621d6b04dff5cdb6169a732fcbdc82024e5b6cff"}
Oct 30 19:37:26.446680 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:26.446292    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:26.469213 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:26.469167    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd" podStartSLOduration=2.178837558 podStartE2EDuration="6.469153627s" totalImagesPullingTime="4.290316069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:26.468110716 +0000 UTC m=+325.383717574" watchObservedRunningTime="2026-10-30 19:37:26.469153627 +0000 UTC m=+325.384760482"
Oct 30 19:37:30.343011 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:30.342984    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-grxfc"
Oct 30 19:37:32.454640 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:32.454613    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:33.311857 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:33.311819    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8fshs"]
Oct 30 19:37:33.318656 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:33.318623    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8fshs"
Oct 30 19:37:33.338337 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:33.338312    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8fshs"]
Oct 30 19:37:33.369106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:33.369069    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gj5tc\""
Oct 30 19:37:33.369106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:33.369086    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 19:37:33.384977 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:33.384945    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9a8a2ef1-9817-46ca-834b-29f53304bc47-tls-cert\") pod \"authorino-6889cbd8c4-8fshs\" (UID: \"9a8a2ef1-9817-46ca-834b-29f53304bc47\") " pod="kuadrant-system/authorino-6889cbd8c4-8fshs"
Oct 30 19:37:33.385208 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:33.385072    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qd6q\" (UniqueName: \"kubernetes.io/projected/9a8a2ef1-9817-46ca-834b-29f53304bc47-kube-api-access-6qd6q\") pod \"authorino-6889cbd8c4-8fshs\" (UID: \"9a8a2ef1-9817-46ca-834b-29f53304bc47\") " pod="kuadrant-system/authorino-6889cbd8c4-8fshs"
Oct 30 19:37:33.486109 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:33.486079    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9a8a2ef1-9817-46ca-834b-29f53304bc47-tls-cert\") pod \"authorino-6889cbd8c4-8fshs\" (UID: \"9a8a2ef1-9817-46ca-834b-29f53304bc47\") " pod="kuadrant-system/authorino-6889cbd8c4-8fshs"
Oct 30 19:37:33.491148 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:33.486166    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qd6q\" (UniqueName: \"kubernetes.io/projected/9a8a2ef1-9817-46ca-834b-29f53304bc47-kube-api-access-6qd6q\") pod \"authorino-6889cbd8c4-8fshs\" (UID: \"9a8a2ef1-9817-46ca-834b-29f53304bc47\") " pod="kuadrant-system/authorino-6889cbd8c4-8fshs"
Oct 30 19:37:33.491148 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:33.488824    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9a8a2ef1-9817-46ca-834b-29f53304bc47-tls-cert\") pod \"authorino-6889cbd8c4-8fshs\" (UID: \"9a8a2ef1-9817-46ca-834b-29f53304bc47\") " pod="kuadrant-system/authorino-6889cbd8c4-8fshs"
Oct 30 19:37:33.499547 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:33.499520    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qd6q\" (UniqueName: \"kubernetes.io/projected/9a8a2ef1-9817-46ca-834b-29f53304bc47-kube-api-access-6qd6q\") pod \"authorino-6889cbd8c4-8fshs\" (UID: \"9a8a2ef1-9817-46ca-834b-29f53304bc47\") " pod="kuadrant-system/authorino-6889cbd8c4-8fshs"
Oct 30 19:37:33.588105 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:33.588021    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8fshs"]
Oct 30 19:37:33.588255 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:33.588198    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8fshs"
Oct 30 19:37:33.861415 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:33.861391    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8fshs"]
Oct 30 19:37:33.863931 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:37:33.863903    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a8a2ef1_9817_46ca_834b_29f53304bc47.slice/crio-b6b6fce15737c171dbbe02edb823eaa7e6ca46a848276f91091220cb8cf9b097 WatchSource:0}: Error finding container b6b6fce15737c171dbbe02edb823eaa7e6ca46a848276f91091220cb8cf9b097: Status 404 returned error can't find the container with id b6b6fce15737c171dbbe02edb823eaa7e6ca46a848276f91091220cb8cf9b097
Oct 30 19:37:34.287477 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:37:34.287439    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:37:34.478468 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:34.478435    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-8fshs" event={"ID":"9a8a2ef1-9817-46ca-834b-29f53304bc47","Type":"ContainerStarted","Data":"b6b6fce15737c171dbbe02edb823eaa7e6ca46a848276f91091220cb8cf9b097"}
Oct 30 19:37:35.484158 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:35.484120    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-8fshs" event={"ID":"9a8a2ef1-9817-46ca-834b-29f53304bc47","Type":"ContainerStarted","Data":"89657a1a67942af4d5b9ffc376e21fa312a1089b44897d55f201474d21014351"}
Oct 30 19:37:35.484606 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:35.484210    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-8fshs" podUID="9a8a2ef1-9817-46ca-834b-29f53304bc47" containerName="authorino" containerID="cri-o://89657a1a67942af4d5b9ffc376e21fa312a1089b44897d55f201474d21014351" gracePeriod=30
Oct 30 19:37:35.508180 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:35.508128    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-8fshs" podStartSLOduration=1.572788048 podStartE2EDuration="2.508112183s" totalImagesPullingTime="935.324135ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:35.507366962 +0000 UTC m=+334.422973821" watchObservedRunningTime="2026-10-30 19:37:35.508112183 +0000 UTC m=+334.423719038"
Oct 30 19:37:35.718627 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:35.718603    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8fshs"
Oct 30 19:37:35.809412 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:35.809325    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8a2ef1-9817-46ca-834b-29f53304bc47-kube-api-access-6qd6q\" (UniqueName: \"kubernetes.io/projected/9a8a2ef1-9817-46ca-834b-29f53304bc47-kube-api-access-6qd6q\") pod \"9a8a2ef1-9817-46ca-834b-29f53304bc47\" (UID: \"9a8a2ef1-9817-46ca-834b-29f53304bc47\") "
Oct 30 19:37:35.809412 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:35.809376    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a8a2ef1-9817-46ca-834b-29f53304bc47-tls-cert\" (UniqueName: \"kubernetes.io/secret/9a8a2ef1-9817-46ca-834b-29f53304bc47-tls-cert\") pod \"9a8a2ef1-9817-46ca-834b-29f53304bc47\" (UID: \"9a8a2ef1-9817-46ca-834b-29f53304bc47\") "
Oct 30 19:37:35.811610 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:35.811577    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8a2ef1-9817-46ca-834b-29f53304bc47-kube-api-access-6qd6q" pod "9a8a2ef1-9817-46ca-834b-29f53304bc47" (UID: "9a8a2ef1-9817-46ca-834b-29f53304bc47"). InnerVolumeSpecName "kube-api-access-6qd6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:35.818690 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:35.818664    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8a2ef1-9817-46ca-834b-29f53304bc47-tls-cert" pod "9a8a2ef1-9817-46ca-834b-29f53304bc47" (UID: "9a8a2ef1-9817-46ca-834b-29f53304bc47"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:35.910289 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:35.910253    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qd6q\" (UniqueName: \"kubernetes.io/projected/9a8a2ef1-9817-46ca-834b-29f53304bc47-kube-api-access-6qd6q\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:35.910289 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:35.910283    6030 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9a8a2ef1-9817-46ca-834b-29f53304bc47-tls-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:36.489738 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:36.489706    6030 generic.go:356] "Generic (PLEG): container finished" podID="9a8a2ef1-9817-46ca-834b-29f53304bc47" containerID="89657a1a67942af4d5b9ffc376e21fa312a1089b44897d55f201474d21014351" exitCode=0
Oct 30 19:37:36.490206 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:36.489749    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-8fshs" event={"ID":"9a8a2ef1-9817-46ca-834b-29f53304bc47","Type":"ContainerDied","Data":"89657a1a67942af4d5b9ffc376e21fa312a1089b44897d55f201474d21014351"}
Oct 30 19:37:36.490206 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:36.489795    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-8fshs" event={"ID":"9a8a2ef1-9817-46ca-834b-29f53304bc47","Type":"ContainerDied","Data":"b6b6fce15737c171dbbe02edb823eaa7e6ca46a848276f91091220cb8cf9b097"}
Oct 30 19:37:36.490206 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:36.489817    6030 scope.go:122] "RemoveContainer" containerID="89657a1a67942af4d5b9ffc376e21fa312a1089b44897d55f201474d21014351"
Oct 30 19:37:36.490206 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:36.489758    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8fshs"
Oct 30 19:37:36.498759 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:36.498738    6030 scope.go:122] "RemoveContainer" containerID="89657a1a67942af4d5b9ffc376e21fa312a1089b44897d55f201474d21014351"
Oct 30 19:37:36.499060 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:37:36.499021    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89657a1a67942af4d5b9ffc376e21fa312a1089b44897d55f201474d21014351\": container with ID starting with 89657a1a67942af4d5b9ffc376e21fa312a1089b44897d55f201474d21014351 not found: ID does not exist" containerID="89657a1a67942af4d5b9ffc376e21fa312a1089b44897d55f201474d21014351"
Oct 30 19:37:36.499105 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:36.499075    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89657a1a67942af4d5b9ffc376e21fa312a1089b44897d55f201474d21014351"} err="failed to get container status \"89657a1a67942af4d5b9ffc376e21fa312a1089b44897d55f201474d21014351\": rpc error: code = NotFound desc = could not find container \"89657a1a67942af4d5b9ffc376e21fa312a1089b44897d55f201474d21014351\": container with ID starting with 89657a1a67942af4d5b9ffc376e21fa312a1089b44897d55f201474d21014351 not found: ID does not exist"
Oct 30 19:37:36.515865 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:36.515837    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8fshs"]
Oct 30 19:37:36.526439 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:36.526414    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8fshs"]
Oct 30 19:37:37.198968 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:37.198926    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8a2ef1-9817-46ca-834b-29f53304bc47" path="/var/lib/kubelet/pods/9a8a2ef1-9817-46ca-834b-29f53304bc47/volumes"
Oct 30 19:37:43.655062 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:43.655021    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"]
Oct 30 19:37:43.655493 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:43.655297    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd" podUID="826c832f-1687-4592-a7d8-a4ce50c48c78" containerName="maas-api" containerID="cri-o://aa7f650666602f2f6eaa52ad621d6b04dff5cdb6169a732fcbdc82024e5b6cff" gracePeriod=30
Oct 30 19:37:43.893617 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:43.893590    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:43.980441 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:43.980354    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826c832f-1687-4592-a7d8-a4ce50c48c78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/826c832f-1687-4592-a7d8-a4ce50c48c78-openshift-service-ca\") pod \"826c832f-1687-4592-a7d8-a4ce50c48c78\" (UID: \"826c832f-1687-4592-a7d8-a4ce50c48c78\") "
Oct 30 19:37:43.980441 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:43.980409    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826c832f-1687-4592-a7d8-a4ce50c48c78-kube-api-access-t6khm\" (UniqueName: \"kubernetes.io/projected/826c832f-1687-4592-a7d8-a4ce50c48c78-kube-api-access-t6khm\") pod \"826c832f-1687-4592-a7d8-a4ce50c48c78\" (UID: \"826c832f-1687-4592-a7d8-a4ce50c48c78\") "
Oct 30 19:37:43.980684 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:43.980467    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/826c832f-1687-4592-a7d8-a4ce50c48c78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/826c832f-1687-4592-a7d8-a4ce50c48c78-metrics-certs\") pod \"826c832f-1687-4592-a7d8-a4ce50c48c78\" (UID: \"826c832f-1687-4592-a7d8-a4ce50c48c78\") "
Oct 30 19:37:43.980684 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:43.980520    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/826c832f-1687-4592-a7d8-a4ce50c48c78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/826c832f-1687-4592-a7d8-a4ce50c48c78-maas-api-tls\") pod \"826c832f-1687-4592-a7d8-a4ce50c48c78\" (UID: \"826c832f-1687-4592-a7d8-a4ce50c48c78\") "
Oct 30 19:37:43.980806 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:43.980785    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826c832f-1687-4592-a7d8-a4ce50c48c78-openshift-service-ca" pod "826c832f-1687-4592-a7d8-a4ce50c48c78" (UID: "826c832f-1687-4592-a7d8-a4ce50c48c78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:43.983007 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:43.982973    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826c832f-1687-4592-a7d8-a4ce50c48c78-kube-api-access-t6khm" pod "826c832f-1687-4592-a7d8-a4ce50c48c78" (UID: "826c832f-1687-4592-a7d8-a4ce50c48c78"). InnerVolumeSpecName "kube-api-access-t6khm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:43.983007 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:43.982989    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826c832f-1687-4592-a7d8-a4ce50c48c78-metrics-certs" pod "826c832f-1687-4592-a7d8-a4ce50c48c78" (UID: "826c832f-1687-4592-a7d8-a4ce50c48c78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:43.983220 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:43.983036    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826c832f-1687-4592-a7d8-a4ce50c48c78-maas-api-tls" pod "826c832f-1687-4592-a7d8-a4ce50c48c78" (UID: "826c832f-1687-4592-a7d8-a4ce50c48c78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:44.081257 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:44.081223    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/826c832f-1687-4592-a7d8-a4ce50c48c78-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:44.081257 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:44.081254    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/826c832f-1687-4592-a7d8-a4ce50c48c78-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:44.081257 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:44.081265    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/826c832f-1687-4592-a7d8-a4ce50c48c78-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:44.081490 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:44.081275    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6khm\" (UniqueName: \"kubernetes.io/projected/826c832f-1687-4592-a7d8-a4ce50c48c78-kube-api-access-t6khm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:44.523023 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:44.522987    6030 generic.go:356] "Generic (PLEG): container finished" podID="826c832f-1687-4592-a7d8-a4ce50c48c78" containerID="aa7f650666602f2f6eaa52ad621d6b04dff5cdb6169a732fcbdc82024e5b6cff" exitCode=0
Oct 30 19:37:44.523247 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:44.523068    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"
Oct 30 19:37:44.523247 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:44.523076    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd" event={"ID":"826c832f-1687-4592-a7d8-a4ce50c48c78","Type":"ContainerDied","Data":"aa7f650666602f2f6eaa52ad621d6b04dff5cdb6169a732fcbdc82024e5b6cff"}
Oct 30 19:37:44.523247 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:44.523137    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd" event={"ID":"826c832f-1687-4592-a7d8-a4ce50c48c78","Type":"ContainerDied","Data":"07099d346f4fa471a485feb1af0c62faaacaae21eb2eded5d4af0ec049da6cd8"}
Oct 30 19:37:44.523247 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:44.523157    6030 scope.go:122] "RemoveContainer" containerID="aa7f650666602f2f6eaa52ad621d6b04dff5cdb6169a732fcbdc82024e5b6cff"
Oct 30 19:37:44.532123 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:44.532104    6030 scope.go:122] "RemoveContainer" containerID="aa7f650666602f2f6eaa52ad621d6b04dff5cdb6169a732fcbdc82024e5b6cff"
Oct 30 19:37:44.532392 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:37:44.532373    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7f650666602f2f6eaa52ad621d6b04dff5cdb6169a732fcbdc82024e5b6cff\": container with ID starting with aa7f650666602f2f6eaa52ad621d6b04dff5cdb6169a732fcbdc82024e5b6cff not found: ID does not exist" containerID="aa7f650666602f2f6eaa52ad621d6b04dff5cdb6169a732fcbdc82024e5b6cff"
Oct 30 19:37:44.532434 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:44.532400    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7f650666602f2f6eaa52ad621d6b04dff5cdb6169a732fcbdc82024e5b6cff"} err="failed to get container status \"aa7f650666602f2f6eaa52ad621d6b04dff5cdb6169a732fcbdc82024e5b6cff\": rpc error: code = NotFound desc = could not find container \"aa7f650666602f2f6eaa52ad621d6b04dff5cdb6169a732fcbdc82024e5b6cff\": container with ID starting with aa7f650666602f2f6eaa52ad621d6b04dff5cdb6169a732fcbdc82024e5b6cff not found: ID does not exist"
Oct 30 19:37:44.556325 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:44.556296    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"]
Oct 30 19:37:44.574602 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:44.574573    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cb9b5d67c-phcbd"]
Oct 30 19:37:45.199205 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:45.199169    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="826c832f-1687-4592-a7d8-a4ce50c48c78" path="/var/lib/kubelet/pods/826c832f-1687-4592-a7d8-a4ce50c48c78/volumes"
Oct 30 19:37:56.729849 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:56.729812    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-l7rcc"]
Oct 30 19:37:56.730263 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:56.730140    6030 cpu_manager.go:414] "containerMap: removing container" podUID="826c832f-1687-4592-a7d8-a4ce50c48c78" containerName="maas-api"
Oct 30 19:37:56.730263 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:56.730151    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826c832f-1687-4592-a7d8-a4ce50c48c78" containerName="maas-api"
Oct 30 19:37:56.730263 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:56.730159    6030 cpu_manager.go:414] "containerMap: removing container" podUID="9a8a2ef1-9817-46ca-834b-29f53304bc47" containerName="authorino"
Oct 30 19:37:56.730263 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:56.730165    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8a2ef1-9817-46ca-834b-29f53304bc47" containerName="authorino"
Oct 30 19:37:56.730263 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:56.730229    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="826c832f-1687-4592-a7d8-a4ce50c48c78" containerName="maas-api"
Oct 30 19:37:56.730263 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:56.730236    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8a2ef1-9817-46ca-834b-29f53304bc47" containerName="authorino"
Oct 30 19:37:56.734202 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:56.734184    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7rcc"
Oct 30 19:37:56.742243 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:56.742223    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 19:37:56.743699 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:56.743680    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-ctgsz\""
Oct 30 19:37:56.746173 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:56.746152    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-l7rcc"]
Oct 30 19:37:56.895389 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:56.895349    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2hqv\" (UniqueName: \"kubernetes.io/projected/2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc-kube-api-access-d2hqv\") pod \"llmisvc-controller-manager-6466c554d8-l7rcc\" (UID: \"2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7rcc"
Oct 30 19:37:56.895565 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:56.895408    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc-cert\") pod \"llmisvc-controller-manager-6466c554d8-l7rcc\" (UID: \"2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7rcc"
Oct 30 19:37:56.996790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:56.996701    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2hqv\" (UniqueName: \"kubernetes.io/projected/2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc-kube-api-access-d2hqv\") pod \"llmisvc-controller-manager-6466c554d8-l7rcc\" (UID: \"2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7rcc"
Oct 30 19:37:56.996790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:56.996755    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc-cert\") pod \"llmisvc-controller-manager-6466c554d8-l7rcc\" (UID: \"2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7rcc"
Oct 30 19:37:56.999355 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:56.999329    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc-cert\") pod \"llmisvc-controller-manager-6466c554d8-l7rcc\" (UID: \"2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7rcc"
Oct 30 19:37:57.010975 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:57.010949    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2hqv\" (UniqueName: \"kubernetes.io/projected/2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc-kube-api-access-d2hqv\") pod \"llmisvc-controller-manager-6466c554d8-l7rcc\" (UID: \"2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7rcc"
Oct 30 19:37:57.046818 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:57.046792    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7rcc"
Oct 30 19:37:57.318349 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:57.318324    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-l7rcc"]
Oct 30 19:37:57.320757 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:37:57.320729    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3f1f39_378f_460c_8b61_b8bfa8f2c2cc.slice/crio-d52334996ff234b4ecbc6c11257895c62a0827a2c8c94baee7048b211fcd28e1 WatchSource:0}: Error finding container d52334996ff234b4ecbc6c11257895c62a0827a2c8c94baee7048b211fcd28e1: Status 404 returned error can't find the container with id d52334996ff234b4ecbc6c11257895c62a0827a2c8c94baee7048b211fcd28e1
Oct 30 19:37:57.574960 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:57.574612    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7rcc" event={"ID":"2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc","Type":"ContainerStarted","Data":"d52334996ff234b4ecbc6c11257895c62a0827a2c8c94baee7048b211fcd28e1"}
Oct 30 19:37:59.584800 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:59.584765    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7rcc" event={"ID":"2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc","Type":"ContainerStarted","Data":"f1b05925f0b935d140ec25ef4a1c11276d59cb4d0d9780edb9a3e2523575a012"}
Oct 30 19:37:59.585258 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:59.584943    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7rcc"
Oct 30 19:37:59.607373 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:37:59.607309    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7rcc" podStartSLOduration=1.903161181 podStartE2EDuration="3.607295963s" totalImagesPullingTime="1.704134782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:59.604990931 +0000 UTC m=+358.520597792" watchObservedRunningTime="2026-10-30 19:37:59.607295963 +0000 UTC m=+358.522902819"
Oct 30 19:38:04.287848 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:38:04.287810    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:38:30.591011 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:38:30.590978    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7rcc"
Oct 30 19:38:34.287475 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:38:34.287433    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:39:04.287282 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:39:04.287239    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:39:17.899953 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:17.899864    6030 generic.go:356] "Generic (PLEG): container finished" podID="c2b7322b-ec51-4436-bedb-b4a923ffdf75" containerID="e3d30d41bb132f6ecc7e468cce8983fae45fa2b10f099341f11309ccee75f834" exitCode=1
Oct 30 19:39:17.899953 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:17.899940    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cf964748-d8zhr" event={"ID":"c2b7322b-ec51-4436-bedb-b4a923ffdf75","Type":"ContainerDied","Data":"e3d30d41bb132f6ecc7e468cce8983fae45fa2b10f099341f11309ccee75f834"}
Oct 30 19:39:17.900364 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:17.900342    6030 scope.go:122] "RemoveContainer" containerID="e3d30d41bb132f6ecc7e468cce8983fae45fa2b10f099341f11309ccee75f834"
Oct 30 19:39:18.906096 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:18.906031    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cf964748-d8zhr" event={"ID":"c2b7322b-ec51-4436-bedb-b4a923ffdf75","Type":"ContainerStarted","Data":"5c8161a100d6ad9d8bd5b85249d7a84bb84ea51ba07df8a4fb90ffd98af58e70"}
Oct 30 19:39:18.906501 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:18.906294    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cf964748-d8zhr"
Oct 30 19:39:29.912790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:29.912757    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cf964748-d8zhr"
Oct 30 19:39:34.286443 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:39:34.286403    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:39:47.007612 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.007528    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"]
Oct 30 19:39:47.011217 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.011189    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.025314 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.025283    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"]
Oct 30 19:39:47.097383 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.097348    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9slkz\""
Oct 30 19:39:47.114545 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.114516    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.114642 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.114552    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.114642 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.114571    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.114727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.114681    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.114770 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.114723    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.114807 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.114761    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j57mw\" (UniqueName: \"kubernetes.io/projected/39855966-df67-4ccf-87fe-c5ec63f95cdf-kube-api-access-j57mw\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.114807 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.114796    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39855966-df67-4ccf-87fe-c5ec63f95cdf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.114892 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.114879    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.114943 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.114932    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.215869 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.215829    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.216087 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.215877    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.216087 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.216003    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57mw\" (UniqueName: \"kubernetes.io/projected/39855966-df67-4ccf-87fe-c5ec63f95cdf-kube-api-access-j57mw\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.216087 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.216073    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39855966-df67-4ccf-87fe-c5ec63f95cdf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.216275 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.216145    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.216275 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.216184    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.216275 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.216207    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.216275 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.216256    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.216470 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.216281    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.216470 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.216307    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.216580 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.216535    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.216637 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.216605    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.216692 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.216658    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.216807 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.216790    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39855966-df67-4ccf-87fe-c5ec63f95cdf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.218612 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.218586    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.218829 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.218809    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.229672 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.229642    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.229805 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.229771    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j57mw\" (UniqueName: \"kubernetes.io/projected/39855966-df67-4ccf-87fe-c5ec63f95cdf-kube-api-access-j57mw\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ldmb6\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.326670 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.326597    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:47.610864 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.610837    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"]
Oct 30 19:39:47.612723 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:39:47.612694    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39855966_df67_4ccf_87fe_c5ec63f95cdf.slice/crio-f7ea23e73f746fe3f23a05b912c289a48bdbab3b436cf862125ca5f3617ba5dd WatchSource:0}: Error finding container f7ea23e73f746fe3f23a05b912c289a48bdbab3b436cf862125ca5f3617ba5dd: Status 404 returned error can't find the container with id f7ea23e73f746fe3f23a05b912c289a48bdbab3b436cf862125ca5f3617ba5dd
Oct 30 19:39:47.614678 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:47.614656    6030 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:39:48.031104 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:48.031072    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6" event={"ID":"39855966-df67-4ccf-87fe-c5ec63f95cdf","Type":"ContainerStarted","Data":"f7ea23e73f746fe3f23a05b912c289a48bdbab3b436cf862125ca5f3617ba5dd"}
Oct 30 19:39:50.560424 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:50.560387    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:39:50.560698 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:50.560468    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:39:50.560698 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:50.560504    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:39:51.044661 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:51.044622    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6" event={"ID":"39855966-df67-4ccf-87fe-c5ec63f95cdf","Type":"ContainerStarted","Data":"ee624a0ef73920fd1f28bae3a91371d753924f1666a945de84a0cd23987a37ba"}
Oct 30 19:39:51.082070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:51.074315    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6" podStartSLOduration=2.129711099 podStartE2EDuration="5.074295039s" totalImagesPullingTime="2.94458394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:51.069264764 +0000 UTC m=+469.984871620" watchObservedRunningTime="2026-10-30 19:39:51.074295039 +0000 UTC m=+469.989901893"
Oct 30 19:39:51.327727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:51.327644    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:51.328901 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:51.328874    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 19:39:51.329029 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:51.328934    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6" podUID="39855966-df67-4ccf-87fe-c5ec63f95cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 19:39:52.327444 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:52.327411    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 19:39:52.327930 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:52.327460    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6" podUID="39855966-df67-4ccf-87fe-c5ec63f95cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 19:39:53.328032 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:53.327987    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 19:39:53.328461 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:53.328074    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6" podUID="39855966-df67-4ccf-87fe-c5ec63f95cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 19:39:54.327746 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:54.327706    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 19:39:54.327924 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:54.327801    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6" podUID="39855966-df67-4ccf-87fe-c5ec63f95cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 19:39:55.327747 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:55.327718    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 19:39:55.328168 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:55.327771    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6" podUID="39855966-df67-4ccf-87fe-c5ec63f95cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 19:39:56.327189 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:56.327154    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 19:39:56.327402 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:56.327210    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6" podUID="39855966-df67-4ccf-87fe-c5ec63f95cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 19:39:57.327929 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:57.327885    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 19:39:57.328352 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:57.327959    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6" podUID="39855966-df67-4ccf-87fe-c5ec63f95cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 19:39:58.752849 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:58.752809    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:58.753349 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:58.753233    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:39:58.753979 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:39:58.753957    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:40:02.121287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.121259    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"]
Oct 30 19:40:02.124735 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.124717    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.159665 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.159637    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948623f8-d637-44f2-a460-43737e3384d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.159927 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.159896    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948623f8-d637-44f2-a460-43737e3384d1-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.160038 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.159942    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948623f8-d637-44f2-a460-43737e3384d1-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.160038 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.159968    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948623f8-d637-44f2-a460-43737e3384d1-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.160038 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.160028    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948623f8-d637-44f2-a460-43737e3384d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.160256 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.160195    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7d6t\" (UniqueName: \"kubernetes.io/projected/948623f8-d637-44f2-a460-43737e3384d1-kube-api-access-c7d6t\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.160334 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.160313    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948623f8-d637-44f2-a460-43737e3384d1-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.160397 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.160364    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948623f8-d637-44f2-a460-43737e3384d1-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.160453 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.160428    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948623f8-d637-44f2-a460-43737e3384d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.161499 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.161478    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"]
Oct 30 19:40:02.261612 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.261575    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948623f8-d637-44f2-a460-43737e3384d1-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.261790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.261623    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948623f8-d637-44f2-a460-43737e3384d1-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.261790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.261644    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948623f8-d637-44f2-a460-43737e3384d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.261790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.261682    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948623f8-d637-44f2-a460-43737e3384d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.261790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.261716    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948623f8-d637-44f2-a460-43737e3384d1-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.261790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.261742    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948623f8-d637-44f2-a460-43737e3384d1-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.262149 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.262123    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948623f8-d637-44f2-a460-43737e3384d1-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.262373 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.262355    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948623f8-d637-44f2-a460-43737e3384d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.262561 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.262519    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7d6t\" (UniqueName: \"kubernetes.io/projected/948623f8-d637-44f2-a460-43737e3384d1-kube-api-access-c7d6t\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.262751 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.262158    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948623f8-d637-44f2-a460-43737e3384d1-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.262843 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.262602    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948623f8-d637-44f2-a460-43737e3384d1-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.262917 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.262183    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948623f8-d637-44f2-a460-43737e3384d1-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.263104 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.263082    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948623f8-d637-44f2-a460-43737e3384d1-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.263178 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.263131    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948623f8-d637-44f2-a460-43737e3384d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.264978 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.264947    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948623f8-d637-44f2-a460-43737e3384d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.265338 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.265315    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948623f8-d637-44f2-a460-43737e3384d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.276122 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.276064    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948623f8-d637-44f2-a460-43737e3384d1-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.276273 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.276256    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7d6t\" (UniqueName: \"kubernetes.io/projected/948623f8-d637-44f2-a460-43737e3384d1-kube-api-access-c7d6t\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-clbw6\" (UID: \"948623f8-d637-44f2-a460-43737e3384d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.437302 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.437264    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:02.713304 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.713260    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"]
Oct 30 19:40:02.715898 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:40:02.715866    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948623f8_d637_44f2_a460_43737e3384d1.slice/crio-092a2e1cfc0053a06ce95eddf5261bfa9da0ceff3eec1d08e8e4e0f1277496cc WatchSource:0}: Error finding container 092a2e1cfc0053a06ce95eddf5261bfa9da0ceff3eec1d08e8e4e0f1277496cc: Status 404 returned error can't find the container with id 092a2e1cfc0053a06ce95eddf5261bfa9da0ceff3eec1d08e8e4e0f1277496cc
Oct 30 19:40:02.718035 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.718000    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:40:02.718195 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.718100    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:40:02.718195 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:02.718153    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:40:03.098121 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:03.098018    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6" event={"ID":"948623f8-d637-44f2-a460-43737e3384d1","Type":"ContainerStarted","Data":"3cf0c6d01815ffe5ef4fea844666fcb76fba3c11d1fd3a7cea69ff3b728f1c61"}
Oct 30 19:40:03.098121 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:03.098093    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6" event={"ID":"948623f8-d637-44f2-a460-43737e3384d1","Type":"ContainerStarted","Data":"092a2e1cfc0053a06ce95eddf5261bfa9da0ceff3eec1d08e8e4e0f1277496cc"}
Oct 30 19:40:03.438152 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:03.438119    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:03.439635 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:03.439609    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-clbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 19:40:03.439753 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:03.439658    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6" podUID="948623f8-d637-44f2-a460-43737e3384d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 19:40:04.304023 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:40:04.303983    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:40:04.438621 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:04.438584    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-clbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 19:40:04.439024 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:04.438645    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6" podUID="948623f8-d637-44f2-a460-43737e3384d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 19:40:05.437792 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:05.437757    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-clbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 19:40:05.437964 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:05.437813    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6" podUID="948623f8-d637-44f2-a460-43737e3384d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 19:40:06.438135 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:06.438090    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-clbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 19:40:06.438529 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:06.438151    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6" podUID="948623f8-d637-44f2-a460-43737e3384d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 19:40:07.438322 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:07.438289    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-clbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 19:40:07.438734 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:07.438353    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6" podUID="948623f8-d637-44f2-a460-43737e3384d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 19:40:08.437748 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:08.437713    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-clbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 19:40:08.437925 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:08.437769    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6" podUID="948623f8-d637-44f2-a460-43737e3384d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 19:40:09.437975 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:09.437934    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-clbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 19:40:09.438396 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:09.437996    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6" podUID="948623f8-d637-44f2-a460-43737e3384d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 19:40:10.895277 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:10.895236    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:10.895750 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:10.895686    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:10.896431 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:10.896411    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6"
Oct 30 19:40:10.924439 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:10.924388    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6" podStartSLOduration=8.924374678 podStartE2EDuration="8.924374678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:03.129533158 +0000 UTC m=+482.045140014" watchObservedRunningTime="2026-10-30 19:40:10.924374678 +0000 UTC m=+489.839981534"
Oct 30 19:40:17.090688 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.090658    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"]
Oct 30 19:40:17.093978 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.093952    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.108819 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.108791    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"]
Oct 30 19:40:17.224471 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.224429    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.224658 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.224475    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.224658 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.224502    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.224754 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.224653    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2j88\" (UniqueName: \"kubernetes.io/projected/715973cd-8d23-4a57-b8ca-d22f81ac95a5-kube-api-access-r2j88\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.224754 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.224699    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.224856 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.224816    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.224914 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.224901    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.224992 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.224974    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.225034 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.225019    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.325605 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.325567    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.325816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.325617    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2j88\" (UniqueName: \"kubernetes.io/projected/715973cd-8d23-4a57-b8ca-d22f81ac95a5-kube-api-access-r2j88\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.325816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.325637    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.325816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.325663    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.325816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.325724    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.325816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.325757    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.325816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.325778    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.325816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.325814    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.326176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.325847    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.326176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.326079    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.326176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.326124    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.326313 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.326293    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.326575 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.326557    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.326766 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.326752    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.328237 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.328210    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.328460 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.328441    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.339290 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.339264    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2j88\" (UniqueName: \"kubernetes.io/projected/715973cd-8d23-4a57-b8ca-d22f81ac95a5-kube-api-access-r2j88\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.341494 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.341432    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-f6lcg\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.407127 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.407087    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:17.701070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.701020    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"]
Oct 30 19:40:17.703318 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:40:17.703285    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715973cd_8d23_4a57_b8ca_d22f81ac95a5.slice/crio-8cc5b0cf6928207bc695f667bf80c0ff9cfdce8a64d8b30d250ab5f2d99f3589 WatchSource:0}: Error finding container 8cc5b0cf6928207bc695f667bf80c0ff9cfdce8a64d8b30d250ab5f2d99f3589: Status 404 returned error can't find the container with id 8cc5b0cf6928207bc695f667bf80c0ff9cfdce8a64d8b30d250ab5f2d99f3589
Oct 30 19:40:17.706174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.706142    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:40:17.706299 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.706224    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:40:17.706299 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:17.706269    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:40:18.167581 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:18.167543    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg" event={"ID":"715973cd-8d23-4a57-b8ca-d22f81ac95a5","Type":"ContainerStarted","Data":"69c922a197baffb753db4989ec1e094ba25d04974711e1b197d67d25a277c6b9"}
Oct 30 19:40:18.168774 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:18.167609    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg" event={"ID":"715973cd-8d23-4a57-b8ca-d22f81ac95a5","Type":"ContainerStarted","Data":"8cc5b0cf6928207bc695f667bf80c0ff9cfdce8a64d8b30d250ab5f2d99f3589"}
Oct 30 19:40:18.409168 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:18.409116    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:18.410802 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:18.410770    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 19:40:18.411036 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:18.411005    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg" podUID="715973cd-8d23-4a57-b8ca-d22f81ac95a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 19:40:19.407978 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:19.407943    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 19:40:19.408447 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:19.408000    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg" podUID="715973cd-8d23-4a57-b8ca-d22f81ac95a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 19:40:20.407810 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:20.407767    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 19:40:20.407985 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:20.407831    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg" podUID="715973cd-8d23-4a57-b8ca-d22f81ac95a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 19:40:21.407423 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:21.407387    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 19:40:21.407899 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:21.407438    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg" podUID="715973cd-8d23-4a57-b8ca-d22f81ac95a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 19:40:22.407880 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:22.407841    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 19:40:22.408336 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:22.407905    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg" podUID="715973cd-8d23-4a57-b8ca-d22f81ac95a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 19:40:22.767850 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:22.767752    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg" podStartSLOduration=5.76773558 podStartE2EDuration="5.76773558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:18.225061869 +0000 UTC m=+497.140668726" watchObservedRunningTime="2026-10-30 19:40:22.76773558 +0000 UTC m=+501.683342437"
Oct 30 19:40:22.768541 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:22.768524    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"]
Oct 30 19:40:22.801876 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:22.801840    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"]
Oct 30 19:40:22.802840 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:22.802328    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:22.812466 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:22.812441    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d892r\""
Oct 30 19:40:22.812602 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:22.812476    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:40:22.812602 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:22.812479    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:40:22.812602 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:22.812514    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b33c04be-serving-cert\""
Oct 30 19:40:22.812602 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:22.812583    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:40:22.915544 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:22.915506    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d09446b2-4a1f-4229-87a8-9805eec2887a-metrics-certs\") pod \"maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t\" (UID: \"d09446b2-4a1f-4229-87a8-9805eec2887a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:22.915745 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:22.915574    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d09446b2-4a1f-4229-87a8-9805eec2887a-openshift-service-ca\") pod \"maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t\" (UID: \"d09446b2-4a1f-4229-87a8-9805eec2887a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:22.915745 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:22.915672    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d09446b2-4a1f-4229-87a8-9805eec2887a-maas-api-tls\") pod \"maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t\" (UID: \"d09446b2-4a1f-4229-87a8-9805eec2887a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:22.915861 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:22.915741    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrn7j\" (UniqueName: \"kubernetes.io/projected/d09446b2-4a1f-4229-87a8-9805eec2887a-kube-api-access-qrn7j\") pod \"maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t\" (UID: \"d09446b2-4a1f-4229-87a8-9805eec2887a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:23.016456 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:23.016416    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d09446b2-4a1f-4229-87a8-9805eec2887a-metrics-certs\") pod \"maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t\" (UID: \"d09446b2-4a1f-4229-87a8-9805eec2887a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:23.016852 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:23.016495    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d09446b2-4a1f-4229-87a8-9805eec2887a-openshift-service-ca\") pod \"maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t\" (UID: \"d09446b2-4a1f-4229-87a8-9805eec2887a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:23.016852 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:23.016518    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d09446b2-4a1f-4229-87a8-9805eec2887a-maas-api-tls\") pod \"maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t\" (UID: \"d09446b2-4a1f-4229-87a8-9805eec2887a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:23.016852 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:23.016559    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrn7j\" (UniqueName: \"kubernetes.io/projected/d09446b2-4a1f-4229-87a8-9805eec2887a-kube-api-access-qrn7j\") pod \"maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t\" (UID: \"d09446b2-4a1f-4229-87a8-9805eec2887a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:23.017426 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:23.017397    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d09446b2-4a1f-4229-87a8-9805eec2887a-openshift-service-ca\") pod \"maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t\" (UID: \"d09446b2-4a1f-4229-87a8-9805eec2887a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:23.019153 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:23.019109    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d09446b2-4a1f-4229-87a8-9805eec2887a-metrics-certs\") pod \"maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t\" (UID: \"d09446b2-4a1f-4229-87a8-9805eec2887a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:23.019228 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:23.019210    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d09446b2-4a1f-4229-87a8-9805eec2887a-maas-api-tls\") pod \"maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t\" (UID: \"d09446b2-4a1f-4229-87a8-9805eec2887a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:23.029733 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:23.029701    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrn7j\" (UniqueName: \"kubernetes.io/projected/d09446b2-4a1f-4229-87a8-9805eec2887a-kube-api-access-qrn7j\") pod \"maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t\" (UID: \"d09446b2-4a1f-4229-87a8-9805eec2887a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:23.117702 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:23.117662    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:23.408298 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:23.407829    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 19:40:23.408298 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:23.407889    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg" podUID="715973cd-8d23-4a57-b8ca-d22f81ac95a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 19:40:23.418321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:23.417767    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"]
Oct 30 19:40:23.858421 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:23.858341    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"]
Oct 30 19:40:23.867898 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:23.867870    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:23.875395 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:23.873452    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"]
Oct 30 19:40:23.928008 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:23.927962    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0897eecc-serving-cert\""
Oct 30 19:40:24.028226 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.028190    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/066d1f28-4316-40df-a2b5-9d2f0481fac4-metrics-certs\") pod \"maas-api-e2e-ait-0897eecc-7d5787c954-62rnp\" (UID: \"066d1f28-4316-40df-a2b5-9d2f0481fac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:24.028965 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.028232    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/066d1f28-4316-40df-a2b5-9d2f0481fac4-openshift-service-ca\") pod \"maas-api-e2e-ait-0897eecc-7d5787c954-62rnp\" (UID: \"066d1f28-4316-40df-a2b5-9d2f0481fac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:24.028965 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.028266    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzqsb\" (UniqueName: \"kubernetes.io/projected/066d1f28-4316-40df-a2b5-9d2f0481fac4-kube-api-access-pzqsb\") pod \"maas-api-e2e-ait-0897eecc-7d5787c954-62rnp\" (UID: \"066d1f28-4316-40df-a2b5-9d2f0481fac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:24.028965 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.028480    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/066d1f28-4316-40df-a2b5-9d2f0481fac4-maas-api-tls\") pod \"maas-api-e2e-ait-0897eecc-7d5787c954-62rnp\" (UID: \"066d1f28-4316-40df-a2b5-9d2f0481fac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:24.129459 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.129414    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/066d1f28-4316-40df-a2b5-9d2f0481fac4-maas-api-tls\") pod \"maas-api-e2e-ait-0897eecc-7d5787c954-62rnp\" (UID: \"066d1f28-4316-40df-a2b5-9d2f0481fac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:24.129459 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.129466    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/066d1f28-4316-40df-a2b5-9d2f0481fac4-metrics-certs\") pod \"maas-api-e2e-ait-0897eecc-7d5787c954-62rnp\" (UID: \"066d1f28-4316-40df-a2b5-9d2f0481fac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:24.129722 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.129492    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/066d1f28-4316-40df-a2b5-9d2f0481fac4-openshift-service-ca\") pod \"maas-api-e2e-ait-0897eecc-7d5787c954-62rnp\" (UID: \"066d1f28-4316-40df-a2b5-9d2f0481fac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:24.129722 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.129513    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqsb\" (UniqueName: \"kubernetes.io/projected/066d1f28-4316-40df-a2b5-9d2f0481fac4-kube-api-access-pzqsb\") pod \"maas-api-e2e-ait-0897eecc-7d5787c954-62rnp\" (UID: \"066d1f28-4316-40df-a2b5-9d2f0481fac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:24.130318 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.130294    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/066d1f28-4316-40df-a2b5-9d2f0481fac4-openshift-service-ca\") pod \"maas-api-e2e-ait-0897eecc-7d5787c954-62rnp\" (UID: \"066d1f28-4316-40df-a2b5-9d2f0481fac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:24.132235 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.132209    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/066d1f28-4316-40df-a2b5-9d2f0481fac4-maas-api-tls\") pod \"maas-api-e2e-ait-0897eecc-7d5787c954-62rnp\" (UID: \"066d1f28-4316-40df-a2b5-9d2f0481fac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:24.132331 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.132292    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/066d1f28-4316-40df-a2b5-9d2f0481fac4-metrics-certs\") pod \"maas-api-e2e-ait-0897eecc-7d5787c954-62rnp\" (UID: \"066d1f28-4316-40df-a2b5-9d2f0481fac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:24.143277 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.143252    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzqsb\" (UniqueName: \"kubernetes.io/projected/066d1f28-4316-40df-a2b5-9d2f0481fac4-kube-api-access-pzqsb\") pod \"maas-api-e2e-ait-0897eecc-7d5787c954-62rnp\" (UID: \"066d1f28-4316-40df-a2b5-9d2f0481fac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:24.187445 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.187410    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:24.232595 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.232552    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t" event={"ID":"d09446b2-4a1f-4229-87a8-9805eec2887a","Type":"ContainerStarted","Data":"872f822eb7b33b60faf9713143ef164fce1b21d6b1dd8c71d479b92320f23f2e"}
Oct 30 19:40:24.232778 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.232614    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t" event={"ID":"d09446b2-4a1f-4229-87a8-9805eec2887a","Type":"ContainerStarted","Data":"7f4d184df496a9c93658b9351434cc40b9796f908e6b113315f95da73a578362"}
Oct 30 19:40:24.232778 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.232758    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:24.257005 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.256922    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t" podStartSLOduration=1.782766394 podStartE2EDuration="2.256903797s" totalImagesPullingTime="474.137403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:24.255725999 +0000 UTC m=+503.171332856" watchObservedRunningTime="2026-10-30 19:40:24.256903797 +0000 UTC m=+503.172510654"
Oct 30 19:40:24.409604 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.408632    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 19:40:24.409604 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.408698    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg" podUID="715973cd-8d23-4a57-b8ca-d22f81ac95a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 19:40:24.465986 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:24.465956    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"]
Oct 30 19:40:24.471291 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:40:24.471263    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066d1f28_4316_40df_a2b5_9d2f0481fac4.slice/crio-adf27407cb06bb520288688ff95d7ac63f30ca80ef18acd412ef8f831fc6fce2 WatchSource:0}: Error finding container adf27407cb06bb520288688ff95d7ac63f30ca80ef18acd412ef8f831fc6fce2: Status 404 returned error can't find the container with id adf27407cb06bb520288688ff95d7ac63f30ca80ef18acd412ef8f831fc6fce2
Oct 30 19:40:25.238768 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:25.238655    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp" event={"ID":"066d1f28-4316-40df-a2b5-9d2f0481fac4","Type":"ContainerStarted","Data":"9f20c2ad5493a008ba136a8f2eedd5b14743eaeb2b8a21e40bd0799446d90fe6"}
Oct 30 19:40:25.238768 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:25.238728    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp" event={"ID":"066d1f28-4316-40df-a2b5-9d2f0481fac4","Type":"ContainerStarted","Data":"adf27407cb06bb520288688ff95d7ac63f30ca80ef18acd412ef8f831fc6fce2"}
Oct 30 19:40:25.239013 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:25.238942    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:25.272616 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:25.272563    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp" podStartSLOduration=1.861206395 podStartE2EDuration="2.2725456s" totalImagesPullingTime="411.339205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:25.271451627 +0000 UTC m=+504.187058485" watchObservedRunningTime="2026-10-30 19:40:25.2725456 +0000 UTC m=+504.188152456"
Oct 30 19:40:26.291930 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:26.291887    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:26.292467 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:26.292313    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:26.293944 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:26.293921    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 19:40:29.771290 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:29.771253    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"]
Oct 30 19:40:29.774936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:29.774914    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:29.790378 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:29.790358    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-d05eeacd-serving-cert\""
Oct 30 19:40:29.800225 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:29.800185    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"]
Oct 30 19:40:29.899261 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:29.899222    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c5bafa5-b83e-4111-a4e4-a22f68dade48-openshift-service-ca\") pod \"maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5\" (UID: \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:29.899437 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:29.899275    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x94n7\" (UniqueName: \"kubernetes.io/projected/6c5bafa5-b83e-4111-a4e4-a22f68dade48-kube-api-access-x94n7\") pod \"maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5\" (UID: \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:29.899437 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:29.899392    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5bafa5-b83e-4111-a4e4-a22f68dade48-metrics-certs\") pod \"maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5\" (UID: \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:29.899437 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:29.899426    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c5bafa5-b83e-4111-a4e4-a22f68dade48-maas-api-tls\") pod \"maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5\" (UID: \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:30.000627 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:30.000594    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c5bafa5-b83e-4111-a4e4-a22f68dade48-openshift-service-ca\") pod \"maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5\" (UID: \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:30.000627 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:30.000635    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94n7\" (UniqueName: \"kubernetes.io/projected/6c5bafa5-b83e-4111-a4e4-a22f68dade48-kube-api-access-x94n7\") pod \"maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5\" (UID: \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:30.000890 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:30.000672    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5bafa5-b83e-4111-a4e4-a22f68dade48-metrics-certs\") pod \"maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5\" (UID: \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:30.000890 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:30.000691    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c5bafa5-b83e-4111-a4e4-a22f68dade48-maas-api-tls\") pod \"maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5\" (UID: \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:30.001278 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:30.001245    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c5bafa5-b83e-4111-a4e4-a22f68dade48-openshift-service-ca\") pod \"maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5\" (UID: \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:30.003264 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:30.003239    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5bafa5-b83e-4111-a4e4-a22f68dade48-metrics-certs\") pod \"maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5\" (UID: \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:30.003369 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:30.003327    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c5bafa5-b83e-4111-a4e4-a22f68dade48-maas-api-tls\") pod \"maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5\" (UID: \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:30.014711 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:30.014684    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x94n7\" (UniqueName: \"kubernetes.io/projected/6c5bafa5-b83e-4111-a4e4-a22f68dade48-kube-api-access-x94n7\") pod \"maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5\" (UID: \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:30.090873 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:30.090794    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:30.245187 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:30.245157    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:30.367530 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:30.367449    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"]
Oct 30 19:40:30.371854 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:40:30.371826    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5bafa5_b83e_4111_a4e4_a22f68dade48.slice/crio-ca34aff42edb5e00cf195957e6bb347367322796184c09221534e386afc1382a WatchSource:0}: Error finding container ca34aff42edb5e00cf195957e6bb347367322796184c09221534e386afc1382a: Status 404 returned error can't find the container with id ca34aff42edb5e00cf195957e6bb347367322796184c09221534e386afc1382a
Oct 30 19:40:31.250682 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:31.250657    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:31.270755 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:31.270726    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5" event={"ID":"6c5bafa5-b83e-4111-a4e4-a22f68dade48","Type":"ContainerStarted","Data":"436b13dfc3af0546891768d492c1d6a80e9232219cf97ed9b692d895938ab672"}
Oct 30 19:40:31.270901 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:31.270771    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5" event={"ID":"6c5bafa5-b83e-4111-a4e4-a22f68dade48","Type":"ContainerStarted","Data":"ca34aff42edb5e00cf195957e6bb347367322796184c09221534e386afc1382a"}
Oct 30 19:40:31.270901 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:31.270793    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:31.296772 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:31.296710    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5" podStartSLOduration=1.8247010769999998 podStartE2EDuration="2.296690668s" totalImagesPullingTime="471.989591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:31.296118418 +0000 UTC m=+510.211725271" watchObservedRunningTime="2026-10-30 19:40:31.296690668 +0000 UTC m=+510.212297527"
Oct 30 19:40:34.288478 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:40:34.288440    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:40:37.281384 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:37.281358    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:49.859534 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:49.859505    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"]
Oct 30 19:40:49.860092 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:49.859759    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t" podUID="d09446b2-4a1f-4229-87a8-9805eec2887a" containerName="maas-api" containerID="cri-o://872f822eb7b33b60faf9713143ef164fce1b21d6b1dd8c71d479b92320f23f2e" gracePeriod=30
Oct 30 19:40:50.098280 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.097854    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"]
Oct 30 19:40:50.098280 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.098127    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp" podUID="066d1f28-4316-40df-a2b5-9d2f0481fac4" containerName="maas-api" containerID="cri-o://9f20c2ad5493a008ba136a8f2eedd5b14743eaeb2b8a21e40bd0799446d90fe6" gracePeriod=30
Oct 30 19:40:50.243527 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.243500    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:50.307461 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.307427    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d09446b2-4a1f-4229-87a8-9805eec2887a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d09446b2-4a1f-4229-87a8-9805eec2887a-openshift-service-ca\") pod \"d09446b2-4a1f-4229-87a8-9805eec2887a\" (UID: \"d09446b2-4a1f-4229-87a8-9805eec2887a\") "
Oct 30 19:40:50.307656 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.307536    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d09446b2-4a1f-4229-87a8-9805eec2887a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d09446b2-4a1f-4229-87a8-9805eec2887a-metrics-certs\") pod \"d09446b2-4a1f-4229-87a8-9805eec2887a\" (UID: \"d09446b2-4a1f-4229-87a8-9805eec2887a\") "
Oct 30 19:40:50.307656 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.307571    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09446b2-4a1f-4229-87a8-9805eec2887a-kube-api-access-qrn7j\" (UniqueName: \"kubernetes.io/projected/d09446b2-4a1f-4229-87a8-9805eec2887a-kube-api-access-qrn7j\") pod \"d09446b2-4a1f-4229-87a8-9805eec2887a\" (UID: \"d09446b2-4a1f-4229-87a8-9805eec2887a\") "
Oct 30 19:40:50.307656 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.307615    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d09446b2-4a1f-4229-87a8-9805eec2887a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d09446b2-4a1f-4229-87a8-9805eec2887a-maas-api-tls\") pod \"d09446b2-4a1f-4229-87a8-9805eec2887a\" (UID: \"d09446b2-4a1f-4229-87a8-9805eec2887a\") "
Oct 30 19:40:50.308910 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.308634    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d09446b2-4a1f-4229-87a8-9805eec2887a-openshift-service-ca" pod "d09446b2-4a1f-4229-87a8-9805eec2887a" (UID: "d09446b2-4a1f-4229-87a8-9805eec2887a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:50.311193 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.311164    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d09446b2-4a1f-4229-87a8-9805eec2887a-metrics-certs" pod "d09446b2-4a1f-4229-87a8-9805eec2887a" (UID: "d09446b2-4a1f-4229-87a8-9805eec2887a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:50.312784 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.312762    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09446b2-4a1f-4229-87a8-9805eec2887a-kube-api-access-qrn7j" pod "d09446b2-4a1f-4229-87a8-9805eec2887a" (UID: "d09446b2-4a1f-4229-87a8-9805eec2887a"). InnerVolumeSpecName "kube-api-access-qrn7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:50.312924 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.312891    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d09446b2-4a1f-4229-87a8-9805eec2887a-maas-api-tls" pod "d09446b2-4a1f-4229-87a8-9805eec2887a" (UID: "d09446b2-4a1f-4229-87a8-9805eec2887a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:50.357876 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.357846    6030 generic.go:356] "Generic (PLEG): container finished" podID="066d1f28-4316-40df-a2b5-9d2f0481fac4" containerID="9f20c2ad5493a008ba136a8f2eedd5b14743eaeb2b8a21e40bd0799446d90fe6" exitCode=0
Oct 30 19:40:50.357993 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.357913    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp" event={"ID":"066d1f28-4316-40df-a2b5-9d2f0481fac4","Type":"ContainerDied","Data":"9f20c2ad5493a008ba136a8f2eedd5b14743eaeb2b8a21e40bd0799446d90fe6"}
Oct 30 19:40:50.359500 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.359475    6030 generic.go:356] "Generic (PLEG): container finished" podID="d09446b2-4a1f-4229-87a8-9805eec2887a" containerID="872f822eb7b33b60faf9713143ef164fce1b21d6b1dd8c71d479b92320f23f2e" exitCode=0
Oct 30 19:40:50.359605 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.359518    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"
Oct 30 19:40:50.359605 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.359544    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t" event={"ID":"d09446b2-4a1f-4229-87a8-9805eec2887a","Type":"ContainerDied","Data":"872f822eb7b33b60faf9713143ef164fce1b21d6b1dd8c71d479b92320f23f2e"}
Oct 30 19:40:50.359605 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.359594    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t" event={"ID":"d09446b2-4a1f-4229-87a8-9805eec2887a","Type":"ContainerDied","Data":"7f4d184df496a9c93658b9351434cc40b9796f908e6b113315f95da73a578362"}
Oct 30 19:40:50.359727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.359625    6030 scope.go:122] "RemoveContainer" containerID="872f822eb7b33b60faf9713143ef164fce1b21d6b1dd8c71d479b92320f23f2e"
Oct 30 19:40:50.366481 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.366464    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:50.374211 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.374187    6030 scope.go:122] "RemoveContainer" containerID="872f822eb7b33b60faf9713143ef164fce1b21d6b1dd8c71d479b92320f23f2e"
Oct 30 19:40:50.374541 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:40:50.374505    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"872f822eb7b33b60faf9713143ef164fce1b21d6b1dd8c71d479b92320f23f2e\": container with ID starting with 872f822eb7b33b60faf9713143ef164fce1b21d6b1dd8c71d479b92320f23f2e not found: ID does not exist" containerID="872f822eb7b33b60faf9713143ef164fce1b21d6b1dd8c71d479b92320f23f2e"
Oct 30 19:40:50.374613 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.374553    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"872f822eb7b33b60faf9713143ef164fce1b21d6b1dd8c71d479b92320f23f2e"} err="failed to get container status \"872f822eb7b33b60faf9713143ef164fce1b21d6b1dd8c71d479b92320f23f2e\": rpc error: code = NotFound desc = could not find container \"872f822eb7b33b60faf9713143ef164fce1b21d6b1dd8c71d479b92320f23f2e\": container with ID starting with 872f822eb7b33b60faf9713143ef164fce1b21d6b1dd8c71d479b92320f23f2e not found: ID does not exist"
Oct 30 19:40:50.409163 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.409132    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d09446b2-4a1f-4229-87a8-9805eec2887a-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:50.409163 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.409161    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d09446b2-4a1f-4229-87a8-9805eec2887a-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:50.409353 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.409174    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrn7j\" (UniqueName: \"kubernetes.io/projected/d09446b2-4a1f-4229-87a8-9805eec2887a-kube-api-access-qrn7j\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:50.409353 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.409182    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d09446b2-4a1f-4229-87a8-9805eec2887a-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:50.454372 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.454338    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"]
Oct 30 19:40:50.457751 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.457723    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b33c04be-5fdb97db46-tj94t"]
Oct 30 19:40:50.510120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.510092    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/066d1f28-4316-40df-a2b5-9d2f0481fac4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/066d1f28-4316-40df-a2b5-9d2f0481fac4-metrics-certs\") pod \"066d1f28-4316-40df-a2b5-9d2f0481fac4\" (UID: \"066d1f28-4316-40df-a2b5-9d2f0481fac4\") "
Oct 30 19:40:50.510319 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.510152    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/066d1f28-4316-40df-a2b5-9d2f0481fac4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/066d1f28-4316-40df-a2b5-9d2f0481fac4-maas-api-tls\") pod \"066d1f28-4316-40df-a2b5-9d2f0481fac4\" (UID: \"066d1f28-4316-40df-a2b5-9d2f0481fac4\") "
Oct 30 19:40:50.510319 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.510181    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/066d1f28-4316-40df-a2b5-9d2f0481fac4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/066d1f28-4316-40df-a2b5-9d2f0481fac4-openshift-service-ca\") pod \"066d1f28-4316-40df-a2b5-9d2f0481fac4\" (UID: \"066d1f28-4316-40df-a2b5-9d2f0481fac4\") "
Oct 30 19:40:50.510319 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.510225    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/066d1f28-4316-40df-a2b5-9d2f0481fac4-kube-api-access-pzqsb\" (UniqueName: \"kubernetes.io/projected/066d1f28-4316-40df-a2b5-9d2f0481fac4-kube-api-access-pzqsb\") pod \"066d1f28-4316-40df-a2b5-9d2f0481fac4\" (UID: \"066d1f28-4316-40df-a2b5-9d2f0481fac4\") "
Oct 30 19:40:50.510666 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.510636    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066d1f28-4316-40df-a2b5-9d2f0481fac4-openshift-service-ca" pod "066d1f28-4316-40df-a2b5-9d2f0481fac4" (UID: "066d1f28-4316-40df-a2b5-9d2f0481fac4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:50.512709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.512681    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066d1f28-4316-40df-a2b5-9d2f0481fac4-kube-api-access-pzqsb" pod "066d1f28-4316-40df-a2b5-9d2f0481fac4" (UID: "066d1f28-4316-40df-a2b5-9d2f0481fac4"). InnerVolumeSpecName "kube-api-access-pzqsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:50.512932 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.512913    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/066d1f28-4316-40df-a2b5-9d2f0481fac4-metrics-certs" pod "066d1f28-4316-40df-a2b5-9d2f0481fac4" (UID: "066d1f28-4316-40df-a2b5-9d2f0481fac4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:50.513098 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.512943    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/066d1f28-4316-40df-a2b5-9d2f0481fac4-maas-api-tls" pod "066d1f28-4316-40df-a2b5-9d2f0481fac4" (UID: "066d1f28-4316-40df-a2b5-9d2f0481fac4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:50.610909 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.610865    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/066d1f28-4316-40df-a2b5-9d2f0481fac4-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:50.610909 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.610909    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/066d1f28-4316-40df-a2b5-9d2f0481fac4-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:50.611159 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.610929    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/066d1f28-4316-40df-a2b5-9d2f0481fac4-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:50.611159 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:50.610951    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzqsb\" (UniqueName: \"kubernetes.io/projected/066d1f28-4316-40df-a2b5-9d2f0481fac4-kube-api-access-pzqsb\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:51.205468 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:51.205392    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d09446b2-4a1f-4229-87a8-9805eec2887a" path="/var/lib/kubelet/pods/d09446b2-4a1f-4229-87a8-9805eec2887a/volumes"
Oct 30 19:40:51.365584 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:51.365557    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"
Oct 30 19:40:51.365769 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:51.365547    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp" event={"ID":"066d1f28-4316-40df-a2b5-9d2f0481fac4","Type":"ContainerDied","Data":"adf27407cb06bb520288688ff95d7ac63f30ca80ef18acd412ef8f831fc6fce2"}
Oct 30 19:40:51.365769 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:51.365710    6030 scope.go:122] "RemoveContainer" containerID="9f20c2ad5493a008ba136a8f2eedd5b14743eaeb2b8a21e40bd0799446d90fe6"
Oct 30 19:40:51.393532 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:51.393496    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"]
Oct 30 19:40:51.397231 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:51.397201    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0897eecc-7d5787c954-62rnp"]
Oct 30 19:40:53.199735 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.199693    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="066d1f28-4316-40df-a2b5-9d2f0481fac4" path="/var/lib/kubelet/pods/066d1f28-4316-40df-a2b5-9d2f0481fac4/volumes"
Oct 30 19:40:53.263924 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.263892    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"]
Oct 30 19:40:53.264470 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.264447    6030 cpu_manager.go:414] "containerMap: removing container" podUID="066d1f28-4316-40df-a2b5-9d2f0481fac4" containerName="maas-api"
Oct 30 19:40:53.264556 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.264471    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="066d1f28-4316-40df-a2b5-9d2f0481fac4" containerName="maas-api"
Oct 30 19:40:53.264556 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.264519    6030 cpu_manager.go:414] "containerMap: removing container" podUID="d09446b2-4a1f-4229-87a8-9805eec2887a" containerName="maas-api"
Oct 30 19:40:53.264556 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.264527    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d09446b2-4a1f-4229-87a8-9805eec2887a" containerName="maas-api"
Oct 30 19:40:53.264684 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.264610    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="066d1f28-4316-40df-a2b5-9d2f0481fac4" containerName="maas-api"
Oct 30 19:40:53.264684 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.264627    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="d09446b2-4a1f-4229-87a8-9805eec2887a" containerName="maas-api"
Oct 30 19:40:53.269548 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.269524    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.284824 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.284788    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"]
Oct 30 19:40:53.287158 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.287132    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2a3e1b-openshift-default-dockercfg-gv7kl\""
Oct 30 19:40:53.337718 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.337678    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-credential-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.337886 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.337737    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-envoy\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.337886 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.337826    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-token\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.338013 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.337892    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-data\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.338013 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.337931    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-workload-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.338013 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.337995    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-podinfo\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.338188 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.338036    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-workload-certs\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.338188 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.338092    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccdec0db-9dd5-4885-b6e0-0583623ad967-istiod-ca-cert\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.338282 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.338203    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t42nz\" (UniqueName: \"kubernetes.io/projected/ccdec0db-9dd5-4885-b6e0-0583623ad967-kube-api-access-t42nz\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.438849 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.438812    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-credential-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.439066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.438883    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-envoy\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.439066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.438909    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-token\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.439066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.438940    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-data\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.439066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.438970    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-workload-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.439066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.439005    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-podinfo\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.439066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.439057    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-workload-certs\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.439370 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.439096    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccdec0db-9dd5-4885-b6e0-0583623ad967-istiod-ca-cert\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.439370 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.439153    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t42nz\" (UniqueName: \"kubernetes.io/projected/ccdec0db-9dd5-4885-b6e0-0583623ad967-kube-api-access-t42nz\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.439370 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.439298    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-credential-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.439625 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.439605    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-workload-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.440150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.440117    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-workload-certs\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.440985 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.440521    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-data\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.441124 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.441091    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccdec0db-9dd5-4885-b6e0-0583623ad967-istiod-ca-cert\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.442211 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.442187    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-envoy\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.443456 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.443435    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-podinfo\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.454488 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.454420    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t42nz\" (UniqueName: \"kubernetes.io/projected/ccdec0db-9dd5-4885-b6e0-0583623ad967-kube-api-access-t42nz\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.457575 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.457548    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-token\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.583273 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.583236    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:53.868899 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.868875    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"]
Oct 30 19:40:53.871407 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:40:53.871363    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccdec0db_9dd5_4885_b6e0_0583623ad967.slice/crio-93cca299a16cdd25f993377c5905887a806e5e7a04e732319d279adaa47807fc WatchSource:0}: Error finding container 93cca299a16cdd25f993377c5905887a806e5e7a04e732319d279adaa47807fc: Status 404 returned error can't find the container with id 93cca299a16cdd25f993377c5905887a806e5e7a04e732319d279adaa47807fc
Oct 30 19:40:53.873580 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.873548    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:40:53.873658 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.873615    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:40:53.873658 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:53.873644    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:40:54.385145 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:54.385096    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t" event={"ID":"ccdec0db-9dd5-4885-b6e0-0583623ad967","Type":"ContainerStarted","Data":"7a2baa1c2bbf44b9136bc0f6aea8da47da956ec360cf61a49f0d6e454b9f9efd"}
Oct 30 19:40:54.385546 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:54.385170    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t" event={"ID":"ccdec0db-9dd5-4885-b6e0-0583623ad967","Type":"ContainerStarted","Data":"93cca299a16cdd25f993377c5905887a806e5e7a04e732319d279adaa47807fc"}
Oct 30 19:40:54.421667 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:54.421622    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t" podStartSLOduration=1.4216064959999999 podStartE2EDuration="1.421606496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:54.42048266 +0000 UTC m=+533.336089517" watchObservedRunningTime="2026-10-30 19:40:54.421606496 +0000 UTC m=+533.337213379"
Oct 30 19:40:54.583735 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:54.583701    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:54.590078 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:54.590036    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:55.400545 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:55.400510    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:55.401696 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:55.401673    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:40:57.686751 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:57.686708    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"]
Oct 30 19:40:57.687219 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:57.686970    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5" podUID="6c5bafa5-b83e-4111-a4e4-a22f68dade48" containerName="maas-api" containerID="cri-o://436b13dfc3af0546891768d492c1d6a80e9232219cf97ed9b692d895938ab672" gracePeriod=30
Oct 30 19:40:57.940721 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:57.940643    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:58.003933 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.003900    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c5bafa5-b83e-4111-a4e4-a22f68dade48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c5bafa5-b83e-4111-a4e4-a22f68dade48-maas-api-tls\") pod \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\" (UID: \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\") "
Oct 30 19:40:58.004163 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.004011    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c5bafa5-b83e-4111-a4e4-a22f68dade48-kube-api-access-x94n7\" (UniqueName: \"kubernetes.io/projected/6c5bafa5-b83e-4111-a4e4-a22f68dade48-kube-api-access-x94n7\") pod \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\" (UID: \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\") "
Oct 30 19:40:58.004163 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.004038    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c5bafa5-b83e-4111-a4e4-a22f68dade48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c5bafa5-b83e-4111-a4e4-a22f68dade48-openshift-service-ca\") pod \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\" (UID: \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\") "
Oct 30 19:40:58.004163 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.004088    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c5bafa5-b83e-4111-a4e4-a22f68dade48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5bafa5-b83e-4111-a4e4-a22f68dade48-metrics-certs\") pod \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\" (UID: \"6c5bafa5-b83e-4111-a4e4-a22f68dade48\") "
Oct 30 19:40:58.004491 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.004467    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5bafa5-b83e-4111-a4e4-a22f68dade48-openshift-service-ca" pod "6c5bafa5-b83e-4111-a4e4-a22f68dade48" (UID: "6c5bafa5-b83e-4111-a4e4-a22f68dade48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:58.006376 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.006342    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5bafa5-b83e-4111-a4e4-a22f68dade48-kube-api-access-x94n7" pod "6c5bafa5-b83e-4111-a4e4-a22f68dade48" (UID: "6c5bafa5-b83e-4111-a4e4-a22f68dade48"). InnerVolumeSpecName "kube-api-access-x94n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:58.006594 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.006565    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5bafa5-b83e-4111-a4e4-a22f68dade48-metrics-certs" pod "6c5bafa5-b83e-4111-a4e4-a22f68dade48" (UID: "6c5bafa5-b83e-4111-a4e4-a22f68dade48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:58.006945 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.006910    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5bafa5-b83e-4111-a4e4-a22f68dade48-maas-api-tls" pod "6c5bafa5-b83e-4111-a4e4-a22f68dade48" (UID: "6c5bafa5-b83e-4111-a4e4-a22f68dade48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:58.104832 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.104802    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x94n7\" (UniqueName: \"kubernetes.io/projected/6c5bafa5-b83e-4111-a4e4-a22f68dade48-kube-api-access-x94n7\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:58.104832 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.104826    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c5bafa5-b83e-4111-a4e4-a22f68dade48-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:58.104832 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.104838    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5bafa5-b83e-4111-a4e4-a22f68dade48-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:58.105103 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.104846    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c5bafa5-b83e-4111-a4e4-a22f68dade48-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:58.419833 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.419784    6030 generic.go:356] "Generic (PLEG): container finished" podID="6c5bafa5-b83e-4111-a4e4-a22f68dade48" containerID="436b13dfc3af0546891768d492c1d6a80e9232219cf97ed9b692d895938ab672" exitCode=0
Oct 30 19:40:58.420121 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.419850    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"
Oct 30 19:40:58.420121 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.419878    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5" event={"ID":"6c5bafa5-b83e-4111-a4e4-a22f68dade48","Type":"ContainerDied","Data":"436b13dfc3af0546891768d492c1d6a80e9232219cf97ed9b692d895938ab672"}
Oct 30 19:40:58.420121 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.419946    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5" event={"ID":"6c5bafa5-b83e-4111-a4e4-a22f68dade48","Type":"ContainerDied","Data":"ca34aff42edb5e00cf195957e6bb347367322796184c09221534e386afc1382a"}
Oct 30 19:40:58.420121 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.419980    6030 scope.go:122] "RemoveContainer" containerID="436b13dfc3af0546891768d492c1d6a80e9232219cf97ed9b692d895938ab672"
Oct 30 19:40:58.430283 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.430265    6030 scope.go:122] "RemoveContainer" containerID="436b13dfc3af0546891768d492c1d6a80e9232219cf97ed9b692d895938ab672"
Oct 30 19:40:58.430564 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:40:58.430546    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"436b13dfc3af0546891768d492c1d6a80e9232219cf97ed9b692d895938ab672\": container with ID starting with 436b13dfc3af0546891768d492c1d6a80e9232219cf97ed9b692d895938ab672 not found: ID does not exist" containerID="436b13dfc3af0546891768d492c1d6a80e9232219cf97ed9b692d895938ab672"
Oct 30 19:40:58.430608 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.430576    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"436b13dfc3af0546891768d492c1d6a80e9232219cf97ed9b692d895938ab672"} err="failed to get container status \"436b13dfc3af0546891768d492c1d6a80e9232219cf97ed9b692d895938ab672\": rpc error: code = NotFound desc = could not find container \"436b13dfc3af0546891768d492c1d6a80e9232219cf97ed9b692d895938ab672\": container with ID starting with 436b13dfc3af0546891768d492c1d6a80e9232219cf97ed9b692d895938ab672 not found: ID does not exist"
Oct 30 19:40:58.450328 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.449284    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"]
Oct 30 19:40:58.482072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:58.464756    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d05eeacd-696f5bdf74-46lf5"]
Oct 30 19:40:59.198688 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.198658    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c5bafa5-b83e-4111-a4e4-a22f68dade48" path="/var/lib/kubelet/pods/6c5bafa5-b83e-4111-a4e4-a22f68dade48/volumes"
Oct 30 19:40:59.581456 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.581376    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"]
Oct 30 19:40:59.581908 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.581893    6030 cpu_manager.go:414] "containerMap: removing container" podUID="6c5bafa5-b83e-4111-a4e4-a22f68dade48" containerName="maas-api"
Oct 30 19:40:59.581963 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.581913    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c5bafa5-b83e-4111-a4e4-a22f68dade48" containerName="maas-api"
Oct 30 19:40:59.582039 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.582029    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c5bafa5-b83e-4111-a4e4-a22f68dade48" containerName="maas-api"
Oct 30 19:40:59.587115 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.587088    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.599987 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.599956    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"]
Oct 30 19:40:59.600197 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.600180    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-7a745c-openshift-default-dockercfg-svrhl\""
Oct 30 19:40:59.720952 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.720493    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.720952 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.720551    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.720952 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.720585    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.720952 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.720620    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d665c73-057a-47b6-9b8c-76768771182f-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.720952 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.720643    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d665c73-057a-47b6-9b8c-76768771182f-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.720952 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.720667    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d665c73-057a-47b6-9b8c-76768771182f-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.720952 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.720798    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd2wk\" (UniqueName: \"kubernetes.io/projected/8d665c73-057a-47b6-9b8c-76768771182f-kube-api-access-wd2wk\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.720952 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.720822    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.720952 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.720847    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.821874 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.821835    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd2wk\" (UniqueName: \"kubernetes.io/projected/8d665c73-057a-47b6-9b8c-76768771182f-kube-api-access-wd2wk\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.821874 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.821875    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.822139 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.821901    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.822139 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.822009    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.822139 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.822086    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.822139 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.822120    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.822353 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.822154    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d665c73-057a-47b6-9b8c-76768771182f-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.822353 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.822179    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d665c73-057a-47b6-9b8c-76768771182f-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.822353 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.822205    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d665c73-057a-47b6-9b8c-76768771182f-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.822353 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.822290    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.822353 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.822330    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.822606 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.822518    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.822872 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.822847    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.822959 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.822911    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d665c73-057a-47b6-9b8c-76768771182f-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.824886 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.824864    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d665c73-057a-47b6-9b8c-76768771182f-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.832590 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.831416    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.848107 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.848071    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd2wk\" (UniqueName: \"kubernetes.io/projected/8d665c73-057a-47b6-9b8c-76768771182f-kube-api-access-wd2wk\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.850723 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.850664    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d665c73-057a-47b6-9b8c-76768771182f-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:40:59.904809 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:40:59.904772    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:41:00.194842 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:00.194816    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"]
Oct 30 19:41:00.197406 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:41:00.197379    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d665c73_057a_47b6_9b8c_76768771182f.slice/crio-05137ba69bd860177a1e8cbbfc07e376961b09e98c67c5ce0db1916adff7aa1f WatchSource:0}: Error finding container 05137ba69bd860177a1e8cbbfc07e376961b09e98c67c5ce0db1916adff7aa1f: Status 404 returned error can't find the container with id 05137ba69bd860177a1e8cbbfc07e376961b09e98c67c5ce0db1916adff7aa1f
Oct 30 19:41:00.199561 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:00.199530    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:41:00.199842 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:00.199600    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:41:00.199842 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:00.199629    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:41:00.432133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:00.432089    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm" event={"ID":"8d665c73-057a-47b6-9b8c-76768771182f","Type":"ContainerStarted","Data":"e0f9e8d327d5ec71302370a739a9c82e1b1ebd4eb442eea6dba5a3ada5bef393"}
Oct 30 19:41:00.432288 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:00.432205    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm" event={"ID":"8d665c73-057a-47b6-9b8c-76768771182f","Type":"ContainerStarted","Data":"05137ba69bd860177a1e8cbbfc07e376961b09e98c67c5ce0db1916adff7aa1f"}
Oct 30 19:41:00.905679 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:00.905644    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:41:01.909958 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:01.909928    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:41:01.944644 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:01.944590    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm" podStartSLOduration=2.944575463 podStartE2EDuration="2.944575463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:00.463896065 +0000 UTC m=+539.379502922" watchObservedRunningTime="2026-10-30 19:41:01.944575463 +0000 UTC m=+540.860182347"
Oct 30 19:41:02.442392 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:02.442039    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:41:02.443195 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:02.443173    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:41:04.288018 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:41:04.287983    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:41:07.368534 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.368503    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"]
Oct 30 19:41:07.379686 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.373810    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.391732 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.391701    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c1fdd6\"/\"kube-root-ca.crt\""
Oct 30 19:41:07.394532 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.394239    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c1fdd6\"/\"openshift-service-ca.crt\""
Oct 30 19:41:07.394532 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.394376    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c1fdd6\"/\"default-dockercfg-qq4v5\""
Oct 30 19:41:07.394532 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.394503    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c1fdd6\"/\"simulator-w3-c1fdd6-kserve-self-signed-certs\""
Oct 30 19:41:07.396708 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.396685    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"]
Oct 30 19:41:07.509764 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.509726    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-home\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.509947 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.509790    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stqr6\" (UniqueName: \"kubernetes.io/projected/b88e4980-4558-4da6-98a5-d59a6fb264eb-kube-api-access-stqr6\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.509947 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.509908    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-tmp-dir\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.510030 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.510012    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-dshm\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.510101 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.510038    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b88e4980-4558-4da6-98a5-d59a6fb264eb-tls-certs\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.510101 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.510087    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-model-cache\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.611575 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.611531    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-home\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.611860 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.611834    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stqr6\" (UniqueName: \"kubernetes.io/projected/b88e4980-4558-4da6-98a5-d59a6fb264eb-kube-api-access-stqr6\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.612003 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.611987    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-tmp-dir\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.612217 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.612199    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-dshm\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.612328 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.612313    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b88e4980-4558-4da6-98a5-d59a6fb264eb-tls-certs\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.612434 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.612422    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-model-cache\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.612966 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.612941    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-model-cache\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.613418 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.613394    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-home\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.614217 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.614196    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-tmp-dir\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.622381 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.622325    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-dshm\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.623956 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.623924    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b88e4980-4558-4da6-98a5-d59a6fb264eb-tls-certs\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.640892 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.640867    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stqr6\" (UniqueName: \"kubernetes.io/projected/b88e4980-4558-4da6-98a5-d59a6fb264eb-kube-api-access-stqr6\") pod \"simulator-w3-c1fdd6-kserve-6874ffc766-vztws\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:07.719495 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:07.719464    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:08.021515 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:08.021482    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"]
Oct 30 19:41:08.024537 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:41:08.024503    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb88e4980_4558_4da6_98a5_d59a6fb264eb.slice/crio-8e48add4cbe3520c0a4670b1e3e474c460195216062bbf2a3ac89a19f9be4acd WatchSource:0}: Error finding container 8e48add4cbe3520c0a4670b1e3e474c460195216062bbf2a3ac89a19f9be4acd: Status 404 returned error can't find the container with id 8e48add4cbe3520c0a4670b1e3e474c460195216062bbf2a3ac89a19f9be4acd
Oct 30 19:41:08.501365 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:08.501327    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" event={"ID":"b88e4980-4558-4da6-98a5-d59a6fb264eb","Type":"ContainerStarted","Data":"8e48add4cbe3520c0a4670b1e3e474c460195216062bbf2a3ac89a19f9be4acd"}
Oct 30 19:41:10.517764 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:10.517724    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" event={"ID":"b88e4980-4558-4da6-98a5-d59a6fb264eb","Type":"ContainerStarted","Data":"6b3bf745e06bad98938d27f2da3f22a5ca67bce3dfa85230385c9b98c9e3ae59"}
Oct 30 19:41:10.541575 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:10.541525    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podStartSLOduration=1.551985088 podStartE2EDuration="3.541512252s" totalImagesPullingTime="1.989527164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:10.540295507 +0000 UTC m=+549.455902363" watchObservedRunningTime="2026-10-30 19:41:10.541512252 +0000 UTC m=+549.457119108"
Oct 30 19:41:10.720246 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:10.720175    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:15.542773 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:15.542739    6030 generic.go:356] "Generic (PLEG): container finished" podID="ccdec0db-9dd5-4885-b6e0-0583623ad967" containerID="7a2baa1c2bbf44b9136bc0f6aea8da47da956ec360cf61a49f0d6e454b9f9efd" exitCode=0
Oct 30 19:41:15.543367 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:15.542814    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t" event={"ID":"ccdec0db-9dd5-4885-b6e0-0583623ad967","Type":"ContainerDied","Data":"7a2baa1c2bbf44b9136bc0f6aea8da47da956ec360cf61a49f0d6e454b9f9efd"}
Oct 30 19:41:15.543367 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:15.543311    6030 scope.go:122] "RemoveContainer" containerID="7a2baa1c2bbf44b9136bc0f6aea8da47da956ec360cf61a49f0d6e454b9f9efd"
Oct 30 19:41:15.544558 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:15.544528    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:41:15.544618 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:15.544599    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:41:15.544652 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:15.544633    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:41:16.564742 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:16.564685    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t" event={"ID":"ccdec0db-9dd5-4885-b6e0-0583623ad967","Type":"ContainerStarted","Data":"7dd902950fb1647ef0caba07bdeff0b410ba9cd97a3a12e28ceb36328f767d5d"}
Oct 30 19:41:16.593071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:16.589884    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:41:16.593448 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:16.593419    6030 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 19:41:16.593633 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:16.593610    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t" podUID="ccdec0db-9dd5-4885-b6e0-0583623ad967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 19:41:17.345429 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.345363    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"]
Oct 30 19:41:17.350606 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.350572    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.365992 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.365870    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f06af6\"/\"simulator-w2-f06af6-kserve-self-signed-certs\""
Oct 30 19:41:17.365992 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.365956    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f06af6\"/\"default-dockercfg-bw24m\""
Oct 30 19:41:17.371197 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.371166    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-tmp-dir\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.371484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.371464    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-tls-certs\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.371779 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.371723    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-dshm\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.371934 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.371919    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-home\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.372243 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.372221    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfrl4\" (UniqueName: \"kubernetes.io/projected/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-kube-api-access-nfrl4\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.372401 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.372385    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-model-cache\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.386966 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.386528    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"]
Oct 30 19:41:17.386966 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.386820    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f06af6\"/\"kube-root-ca.crt\""
Oct 30 19:41:17.386966 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.386931    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f06af6\"/\"openshift-service-ca.crt\""
Oct 30 19:41:17.478611 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.478566    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-dshm\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.478880 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.478857    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-home\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.479019 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.479003    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfrl4\" (UniqueName: \"kubernetes.io/projected/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-kube-api-access-nfrl4\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.479198 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.479181    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-model-cache\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.503110 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.479354    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-tmp-dir\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.503420 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.503393    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-tls-certs\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.526531 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.526498    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-home\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.541428 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.527014    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-model-cache\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.541648 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.527292    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-tmp-dir\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.541747 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.539145    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-tls-certs\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.541838 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.539385    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-dshm\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.542571 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.542520    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfrl4\" (UniqueName: \"kubernetes.io/projected/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-kube-api-access-nfrl4\") pod \"simulator-w2-f06af6-kserve-56c56c6d5b-f29fj\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") " pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.592186 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.592129    6030 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 19:41:17.592711 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.592194    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t" podUID="ccdec0db-9dd5-4885-b6e0-0583623ad967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 19:41:17.667274 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.667235    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:17.727068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.724164    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:17.759266 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:17.759237    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:18.084319 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:18.079434    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"]
Oct 30 19:41:18.586510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:18.586471    6030 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 19:41:18.586838 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:18.586806    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t" podUID="ccdec0db-9dd5-4885-b6e0-0583623ad967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 19:41:18.612318 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:18.612282    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" event={"ID":"2d6f9a7a-40c2-4169-b82a-48cd170dd8db","Type":"ContainerStarted","Data":"f25e78f30d5324cbc95da380ecf11defe928c936e3d65e412adc84d099756d59"}
Oct 30 19:41:18.612875 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:18.612852    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" event={"ID":"2d6f9a7a-40c2-4169-b82a-48cd170dd8db","Type":"ContainerStarted","Data":"1163cc1337e75daf7ee1fea333498db7cbcb29c2c6c266db6e61f3fcd159792e"}
Oct 30 19:41:18.634830 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:18.634781    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podStartSLOduration=1.634767939 podStartE2EDuration="1.634767939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:18.634331317 +0000 UTC m=+557.549938167" watchObservedRunningTime="2026-10-30 19:41:18.634767939 +0000 UTC m=+557.550374795"
Oct 30 19:41:18.638820 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:18.638793    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:41:18.668114 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:18.668082    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:19.589071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:19.588617    6030 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 19:41:19.589071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:19.588697    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t" podUID="ccdec0db-9dd5-4885-b6e0-0583623ad967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 19:41:20.596074 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:20.595732    6030 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 19:41:20.596074 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:20.595792    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t" podUID="ccdec0db-9dd5-4885-b6e0-0583623ad967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 19:41:21.583750 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:21.583711    6030 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 19:41:21.583970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:21.583793    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t" podUID="ccdec0db-9dd5-4885-b6e0-0583623ad967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 19:41:22.589612 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:22.589577    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:41:22.590151 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:22.589910    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:41:22.590775 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:22.590754    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 19:41:25.670310 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:25.670281    6030 generic.go:356] "Generic (PLEG): container finished" podID="8d665c73-057a-47b6-9b8c-76768771182f" containerID="e0f9e8d327d5ec71302370a739a9c82e1b1ebd4eb442eea6dba5a3ada5bef393" exitCode=0
Oct 30 19:41:25.670780 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:25.670368    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm" event={"ID":"8d665c73-057a-47b6-9b8c-76768771182f","Type":"ContainerDied","Data":"e0f9e8d327d5ec71302370a739a9c82e1b1ebd4eb442eea6dba5a3ada5bef393"}
Oct 30 19:41:25.670840 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:25.670815    6030 scope.go:122] "RemoveContainer" containerID="e0f9e8d327d5ec71302370a739a9c82e1b1ebd4eb442eea6dba5a3ada5bef393"
Oct 30 19:41:25.672138 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:25.672108    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:41:25.672208 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:25.672177    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:41:25.672242 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:25.672208    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:41:26.739397 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:26.739349    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm" event={"ID":"8d665c73-057a-47b6-9b8c-76768771182f","Type":"ContainerStarted","Data":"1f7a34ebe781cd6099e72fd2f6bbca97eccab79d3d0994cb097f78059be91132"}
Oct 30 19:41:26.906088 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:26.905625    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:41:26.906294 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:26.906246    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:41:26.906382 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:26.906306    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm" podUID="8d665c73-057a-47b6-9b8c-76768771182f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 19:41:27.668402 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:27.668360    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:27.703470 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:27.703440    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:27.756229 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:27.756200    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:41:27.906143 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:27.906106    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:41:27.906302 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:27.906169    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm" podUID="8d665c73-057a-47b6-9b8c-76768771182f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 19:41:28.905307 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:28.905270    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:41:28.905703 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:28.905326    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm" podUID="8d665c73-057a-47b6-9b8c-76768771182f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 19:41:29.906195 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:29.906157    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:41:29.906596 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:29.906227    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm" podUID="8d665c73-057a-47b6-9b8c-76768771182f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 19:41:30.906170 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:30.906135    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:41:30.906346 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:30.906192    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm" podUID="8d665c73-057a-47b6-9b8c-76768771182f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 19:41:31.909880 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:31.909848    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:41:31.910372 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:31.910179    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:41:31.910996 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:31.910979    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:41:34.067542 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.067513    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"]
Oct 30 19:41:34.071351 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.071331    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.086613 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.086590    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8836a8\"/\"simulator-w0-8836a8-kserve-self-signed-certs\""
Oct 30 19:41:34.092998 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.092970    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8836a8\"/\"default-dockercfg-vp47g\""
Oct 30 19:41:34.095914 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.095892    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"]
Oct 30 19:41:34.096913 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.096898    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8836a8\"/\"kube-root-ca.crt\""
Oct 30 19:41:34.097026 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.097013    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8836a8\"/\"openshift-service-ca.crt\""
Oct 30 19:41:34.256893 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.256855    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-home\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.257066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.256899    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-model-cache\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.257066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.256984    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdc2s\" (UniqueName: \"kubernetes.io/projected/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-kube-api-access-rdc2s\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.257066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.257026    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-tls-certs\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.257214 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.257173    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-dshm\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.257214 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.257195    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-tmp-dir\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.287691 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:41:34.287637    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:41:34.364125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.360099    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdc2s\" (UniqueName: \"kubernetes.io/projected/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-kube-api-access-rdc2s\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.364125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.360165    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-tls-certs\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.364125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.360253    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-dshm\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.364125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.360282    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-tmp-dir\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.364125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.360354    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-home\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.364125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.360389    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-model-cache\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.364125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.360839    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-model-cache\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.364125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.361984    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-tmp-dir\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.364125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.363211    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-home\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.380107 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.379261    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-dshm\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.384769 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.384738    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-tls-certs\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.386242 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.386222    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdc2s\" (UniqueName: \"kubernetes.io/projected/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-kube-api-access-rdc2s\") pod \"simulator-w0-8836a8-kserve-689b4899f-p4xcv\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") " pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.690433 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.690310    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:34.981266 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:34.981241    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"]
Oct 30 19:41:34.983764 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:41:34.983733    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19a0c0d2_60d9_4d3c_b88b_bdd5a661f545.slice/crio-f2f2985343fb3cb215349e41bd62e6d564a18d813a89b6e7350f5ba0109b4777 WatchSource:0}: Error finding container f2f2985343fb3cb215349e41bd62e6d564a18d813a89b6e7350f5ba0109b4777: Status 404 returned error can't find the container with id f2f2985343fb3cb215349e41bd62e6d564a18d813a89b6e7350f5ba0109b4777
Oct 30 19:41:35.789968 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:35.789920    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" event={"ID":"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545","Type":"ContainerStarted","Data":"348340ebe6ce66dfc18a6abef92c645d950010ff86e1cbdfbc5030401b915906"}
Oct 30 19:41:35.790407 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:35.790023    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" event={"ID":"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545","Type":"ContainerStarted","Data":"f2f2985343fb3cb215349e41bd62e6d564a18d813a89b6e7350f5ba0109b4777"}
Oct 30 19:41:35.812220 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:35.812171    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podStartSLOduration=1.8121580229999998 podStartE2EDuration="1.812158023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:35.810904175 +0000 UTC m=+574.726511031" watchObservedRunningTime="2026-10-30 19:41:35.812158023 +0000 UTC m=+574.727764906"
Oct 30 19:41:36.311910 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.311879    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"]
Oct 30 19:41:36.315585 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.315563    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.331374 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.331342    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"]
Oct 30 19:41:36.334504 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.334449    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7a745c\"/\"default-dockercfg-zlh4x\""
Oct 30 19:41:36.335295 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.335274    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7a745c\"/\"simulator-w1-7a745c-kserve-self-signed-certs\""
Oct 30 19:41:36.335405 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.335358    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7a745c\"/\"kube-root-ca.crt\""
Oct 30 19:41:36.335452 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.335413    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7a745c\"/\"openshift-service-ca.crt\""
Oct 30 19:41:36.391975 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.391903    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-tmp-dir\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.391975 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.391963    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-home\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.392265 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.392005    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-dshm\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.392265 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.392131    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwhmw\" (UniqueName: \"kubernetes.io/projected/722a2823-1014-4139-8a7b-d6d6e70d1d2e-kube-api-access-qwhmw\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.392265 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.392236    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722a2823-1014-4139-8a7b-d6d6e70d1d2e-tls-certs\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.392433 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.392287    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-model-cache\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.493538 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.493492    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwhmw\" (UniqueName: \"kubernetes.io/projected/722a2823-1014-4139-8a7b-d6d6e70d1d2e-kube-api-access-qwhmw\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.493727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.493553    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722a2823-1014-4139-8a7b-d6d6e70d1d2e-tls-certs\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.493727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.493597    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-model-cache\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.493727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.493665    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-tmp-dir\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.493727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.493689    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-home\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.493727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.493713    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-dshm\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.494188 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.494162    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-tmp-dir\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.494298 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.494191    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-model-cache\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.494681 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.494648    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-home\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.499478 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.497891    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-dshm\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.499787 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.499763    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722a2823-1014-4139-8a7b-d6d6e70d1d2e-tls-certs\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.532067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.531930    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwhmw\" (UniqueName: \"kubernetes.io/projected/722a2823-1014-4139-8a7b-d6d6e70d1d2e-kube-api-access-qwhmw\") pod \"simulator-w1-7a745c-kserve-99c68d58f-ghctf\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") " pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.628106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.628074    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:36.691385 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.691351    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:36.905090 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:36.905065    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"]
Oct 30 19:41:36.906759 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:41:36.906724    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod722a2823_1014_4139_8a7b_d6d6e70d1d2e.slice/crio-a9f7ae371b7cc15687f730481d4c319e5e2db70bad9fcf1fc4c2f85aab6436a8 WatchSource:0}: Error finding container a9f7ae371b7cc15687f730481d4c319e5e2db70bad9fcf1fc4c2f85aab6436a8: Status 404 returned error can't find the container with id a9f7ae371b7cc15687f730481d4c319e5e2db70bad9fcf1fc4c2f85aab6436a8
Oct 30 19:41:37.812288 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:37.812247    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" event={"ID":"722a2823-1014-4139-8a7b-d6d6e70d1d2e","Type":"ContainerStarted","Data":"3ec47560be88eb0e29f246ed59016796cc680a2ee5f9c503f3447c4eaf9d8f5c"}
Oct 30 19:41:37.812487 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:37.812301    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" event={"ID":"722a2823-1014-4139-8a7b-d6d6e70d1d2e","Type":"ContainerStarted","Data":"a9f7ae371b7cc15687f730481d4c319e5e2db70bad9fcf1fc4c2f85aab6436a8"}
Oct 30 19:41:37.835092 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:37.835021    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podStartSLOduration=1.8350075750000001 podStartE2EDuration="1.835007575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:37.834281078 +0000 UTC m=+576.749887935" watchObservedRunningTime="2026-10-30 19:41:37.835007575 +0000 UTC m=+576.750614432"
Oct 30 19:41:38.628452 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:38.628424    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:42.241594 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.241553    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"]
Oct 30 19:41:42.245287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.245263    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.255169 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.255148    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f06af6\"/\"premium-w2-f06af6-kserve-self-signed-certs\""
Oct 30 19:41:42.258616 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.258591    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"]
Oct 30 19:41:42.360597 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.360554    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f531fa87-3d67-4d59-8c6e-9869a8852459-tls-certs\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.360767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.360642    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-model-cache\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.360767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.360726    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-home\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.360843 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.360798    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pxs7\" (UniqueName: \"kubernetes.io/projected/f531fa87-3d67-4d59-8c6e-9869a8852459-kube-api-access-4pxs7\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.360843 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.360831    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-dshm\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.360905 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.360863    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-tmp-dir\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.461666 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.461628    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f531fa87-3d67-4d59-8c6e-9869a8852459-tls-certs\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.461866 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.461680    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-model-cache\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.461866 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.461698    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-home\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.461866 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.461725    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pxs7\" (UniqueName: \"kubernetes.io/projected/f531fa87-3d67-4d59-8c6e-9869a8852459-kube-api-access-4pxs7\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.461866 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.461748    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-dshm\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.461866 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.461769    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-tmp-dir\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.462213 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.462180    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-home\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.462332 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.462180    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-model-cache\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.462332 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.462317    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-tmp-dir\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.464280 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.464259    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-dshm\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.464430 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.464413    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f531fa87-3d67-4d59-8c6e-9869a8852459-tls-certs\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.477437 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.477410    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pxs7\" (UniqueName: \"kubernetes.io/projected/f531fa87-3d67-4d59-8c6e-9869a8852459-kube-api-access-4pxs7\") pod \"premium-w2-f06af6-kserve-64d7d659c-gkw67\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") " pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.566038 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.565958    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:42.877860 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:42.877825    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"]
Oct 30 19:41:42.882353 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:41:42.882326    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf531fa87_3d67_4d59_8c6e_9869a8852459.slice/crio-9cba54c5c77a7ce6e3171ed8d94d4a3b66d579069172817db769bb224f4d8bc9 WatchSource:0}: Error finding container 9cba54c5c77a7ce6e3171ed8d94d4a3b66d579069172817db769bb224f4d8bc9: Status 404 returned error can't find the container with id 9cba54c5c77a7ce6e3171ed8d94d4a3b66d579069172817db769bb224f4d8bc9
Oct 30 19:41:43.842487 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:43.842454    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" event={"ID":"f531fa87-3d67-4d59-8c6e-9869a8852459","Type":"ContainerStarted","Data":"042978397fd8ec66811157de161746daa23ab0563f53bdc5a109f693f9a557ff"}
Oct 30 19:41:43.842901 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:43.842494    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" event={"ID":"f531fa87-3d67-4d59-8c6e-9869a8852459","Type":"ContainerStarted","Data":"9cba54c5c77a7ce6e3171ed8d94d4a3b66d579069172817db769bb224f4d8bc9"}
Oct 30 19:41:43.864991 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:43.864929    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podStartSLOduration=1.864910107 podStartE2EDuration="1.864910107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:43.864182221 +0000 UTC m=+582.779789088" watchObservedRunningTime="2026-10-30 19:41:43.864910107 +0000 UTC m=+582.780516964"
Oct 30 19:41:44.566484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:44.566453    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:44.691312 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:44.691281    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:44.704722 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:44.704689    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:44.858747 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:44.858662    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:41:46.629198 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:46.629169    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:46.641759 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:46.641736    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:46.868884 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:46.868852    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:41:52.566479 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:52.566441    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:52.580311 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:52.580286    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:52.899983 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:52.899949    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:41:54.927176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:54.927141    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"]
Oct 30 19:41:54.931500 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:54.931479    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:54.942334 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:54.942310    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8836a8\"/\"premium-w0-8836a8-kserve-self-signed-certs\""
Oct 30 19:41:54.944639 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:54.944609    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"]
Oct 30 19:41:55.024325 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.024285    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-model-cache\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.024518 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.024394    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j8d6\" (UniqueName: \"kubernetes.io/projected/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-kube-api-access-8j8d6\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.024593 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.024570    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-tmp-dir\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.024638 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.024619    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-tls-certs\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.024708 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.024690    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-home\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.024775 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.024743    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-dshm\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.126020 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.125979    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j8d6\" (UniqueName: \"kubernetes.io/projected/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-kube-api-access-8j8d6\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.126213 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.126140    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-tmp-dir\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.126213 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.126180    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-tls-certs\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.126286 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.126214    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-home\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.126286 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.126243    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-dshm\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.126351 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.126306    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-model-cache\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.126847 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.126815    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-model-cache\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.126973 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.126852    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-tmp-dir\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.126973 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.126880    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-home\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.128963 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.128935    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-dshm\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.129401 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.129379    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-tls-certs\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.140141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.140116    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j8d6\" (UniqueName: \"kubernetes.io/projected/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-kube-api-access-8j8d6\") pod \"premium-w0-8836a8-kserve-68bbcf8f58-nxpg5\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") " pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.248257 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.248176    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:41:55.526677 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.526655    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"]
Oct 30 19:41:55.529097 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:41:55.529068    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8063f3b_1086_4b3e_9364_b0e9a5f8bfc9.slice/crio-922f1879fbe19d19d75e5e9bce437316f4b7a81b71cc6cfa3808e8d48005d925 WatchSource:0}: Error finding container 922f1879fbe19d19d75e5e9bce437316f4b7a81b71cc6cfa3808e8d48005d925: Status 404 returned error can't find the container with id 922f1879fbe19d19d75e5e9bce437316f4b7a81b71cc6cfa3808e8d48005d925
Oct 30 19:41:55.907419 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.907384    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" event={"ID":"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9","Type":"ContainerStarted","Data":"9645fdfaa19212d346da07306e1aba1e5bdaa61c888d57880b9601877d3aa98b"}
Oct 30 19:41:55.907593 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.907428    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" event={"ID":"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9","Type":"ContainerStarted","Data":"922f1879fbe19d19d75e5e9bce437316f4b7a81b71cc6cfa3808e8d48005d925"}
Oct 30 19:41:55.929217 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:55.929152    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podStartSLOduration=1.929134829 podStartE2EDuration="1.929134829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:55.929017605 +0000 UTC m=+594.844624464" watchObservedRunningTime="2026-10-30 19:41:55.929134829 +0000 UTC m=+594.844741689"
Oct 30 19:41:56.249108 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:41:56.248992    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:42:04.289292 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:42:04.289250    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:42:05.248440 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.248408    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:42:05.260835 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.260812    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:42:05.570027 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.569940    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"]
Oct 30 19:42:05.574161 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.574136    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"
Oct 30 19:42:05.581434 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.581410    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:05.581574 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.581429    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:42:05.581574 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.581502    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:42:05.581725 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.581620    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 19:42:05.586515 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.586476    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"]
Oct 30 19:42:05.651445 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.651411    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea1208b-a0f4-4882-9da5-7e85331975b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb905a7b-mnj7w\" (UID: \"aea1208b-a0f4-4882-9da5-7e85331975b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"
Oct 30 19:42:05.651598 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.651462    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea1208b-a0f4-4882-9da5-7e85331975b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb905a7b-mnj7w\" (UID: \"aea1208b-a0f4-4882-9da5-7e85331975b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"
Oct 30 19:42:05.651598 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.651549    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrsc\" (UniqueName: \"kubernetes.io/projected/aea1208b-a0f4-4882-9da5-7e85331975b3-kube-api-access-mhrsc\") pod \"maas-api-revoke-sub-eb905a7b-mnj7w\" (UID: \"aea1208b-a0f4-4882-9da5-7e85331975b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"
Oct 30 19:42:05.769370 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.769331    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea1208b-a0f4-4882-9da5-7e85331975b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb905a7b-mnj7w\" (UID: \"aea1208b-a0f4-4882-9da5-7e85331975b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"
Oct 30 19:42:05.769661 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.769638    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea1208b-a0f4-4882-9da5-7e85331975b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb905a7b-mnj7w\" (UID: \"aea1208b-a0f4-4882-9da5-7e85331975b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"
Oct 30 19:42:05.769798 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.769785    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrsc\" (UniqueName: \"kubernetes.io/projected/aea1208b-a0f4-4882-9da5-7e85331975b3-kube-api-access-mhrsc\") pod \"maas-api-revoke-sub-eb905a7b-mnj7w\" (UID: \"aea1208b-a0f4-4882-9da5-7e85331975b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"
Oct 30 19:42:05.771127 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.771094    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea1208b-a0f4-4882-9da5-7e85331975b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb905a7b-mnj7w\" (UID: \"aea1208b-a0f4-4882-9da5-7e85331975b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"
Oct 30 19:42:05.777478 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.777448    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea1208b-a0f4-4882-9da5-7e85331975b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb905a7b-mnj7w\" (UID: \"aea1208b-a0f4-4882-9da5-7e85331975b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"
Oct 30 19:42:05.789685 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.789651    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrsc\" (UniqueName: \"kubernetes.io/projected/aea1208b-a0f4-4882-9da5-7e85331975b3-kube-api-access-mhrsc\") pod \"maas-api-revoke-sub-eb905a7b-mnj7w\" (UID: \"aea1208b-a0f4-4882-9da5-7e85331975b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"
Oct 30 19:42:05.889764 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.889730    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"
Oct 30 19:42:05.971495 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:05.971446    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:42:06.163890 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:06.163866    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"]
Oct 30 19:42:06.165534 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:42:06.165505    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea1208b_a0f4_4882_9da5_7e85331975b3.slice/crio-f3f125bd4dd12a603f9f3e5dc167e506683fc2a55538910f3e47a2e1f1fa97c9 WatchSource:0}: Error finding container f3f125bd4dd12a603f9f3e5dc167e506683fc2a55538910f3e47a2e1f1fa97c9: Status 404 returned error can't find the container with id f3f125bd4dd12a603f9f3e5dc167e506683fc2a55538910f3e47a2e1f1fa97c9
Oct 30 19:42:06.963629 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:06.963596    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w" event={"ID":"aea1208b-a0f4-4882-9da5-7e85331975b3","Type":"ContainerStarted","Data":"f3f125bd4dd12a603f9f3e5dc167e506683fc2a55538910f3e47a2e1f1fa97c9"}
Oct 30 19:42:09.980812 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:09.980771    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w" event={"ID":"aea1208b-a0f4-4882-9da5-7e85331975b3","Type":"ContainerStarted","Data":"cc3658058ee347e1cbe956df9e06f5ce45b759a12bae92f3967e16335fe3c708"}
Oct 30 19:42:10.005629 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:10.005571    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w" podStartSLOduration=1.9309404460000001 podStartE2EDuration="5.005556716s" totalImagesPullingTime="3.07461627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:10.004849332 +0000 UTC m=+608.920456188" watchObservedRunningTime="2026-10-30 19:42:10.005556716 +0000 UTC m=+608.921163569"
Oct 30 19:42:10.987552 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:10.987518    6030 generic.go:356] "Generic (PLEG): container finished" podID="aea1208b-a0f4-4882-9da5-7e85331975b3" containerID="cc3658058ee347e1cbe956df9e06f5ce45b759a12bae92f3967e16335fe3c708" exitCode=0
Oct 30 19:42:10.987998 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:10.987559    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w" event={"ID":"aea1208b-a0f4-4882-9da5-7e85331975b3","Type":"ContainerDied","Data":"cc3658058ee347e1cbe956df9e06f5ce45b759a12bae92f3967e16335fe3c708"}
Oct 30 19:42:12.128006 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:12.127982    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"
Oct 30 19:42:12.245580 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:12.245485    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea1208b-a0f4-4882-9da5-7e85331975b3-kube-api-access-mhrsc\" (UniqueName: \"kubernetes.io/projected/aea1208b-a0f4-4882-9da5-7e85331975b3-kube-api-access-mhrsc\") pod \"aea1208b-a0f4-4882-9da5-7e85331975b3\" (UID: \"aea1208b-a0f4-4882-9da5-7e85331975b3\") "
Oct 30 19:42:12.245738 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:12.245624    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aea1208b-a0f4-4882-9da5-7e85331975b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea1208b-a0f4-4882-9da5-7e85331975b3-maas-api-openshift-ca\") pod \"aea1208b-a0f4-4882-9da5-7e85331975b3\" (UID: \"aea1208b-a0f4-4882-9da5-7e85331975b3\") "
Oct 30 19:42:12.245738 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:12.245652    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aea1208b-a0f4-4882-9da5-7e85331975b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea1208b-a0f4-4882-9da5-7e85331975b3-maas-api-xks-ca\") pod \"aea1208b-a0f4-4882-9da5-7e85331975b3\" (UID: \"aea1208b-a0f4-4882-9da5-7e85331975b3\") "
Oct 30 19:42:12.245989 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:12.245965    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea1208b-a0f4-4882-9da5-7e85331975b3-maas-api-openshift-ca" pod "aea1208b-a0f4-4882-9da5-7e85331975b3" (UID: "aea1208b-a0f4-4882-9da5-7e85331975b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:12.247929 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:12.247903    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aea1208b-a0f4-4882-9da5-7e85331975b3-maas-api-xks-ca" pod "aea1208b-a0f4-4882-9da5-7e85331975b3" (UID: "aea1208b-a0f4-4882-9da5-7e85331975b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:12.248126 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:12.248110    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea1208b-a0f4-4882-9da5-7e85331975b3-kube-api-access-mhrsc" pod "aea1208b-a0f4-4882-9da5-7e85331975b3" (UID: "aea1208b-a0f4-4882-9da5-7e85331975b3"). InnerVolumeSpecName "kube-api-access-mhrsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:12.346884 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:12.346846    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhrsc\" (UniqueName: \"kubernetes.io/projected/aea1208b-a0f4-4882-9da5-7e85331975b3-kube-api-access-mhrsc\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:12.346884 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:12.346882    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea1208b-a0f4-4882-9da5-7e85331975b3-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:12.346884 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:12.346892    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea1208b-a0f4-4882-9da5-7e85331975b3-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:13.001585 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:13.001543    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"
Oct 30 19:42:13.002132 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:13.001579    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w" event={"ID":"aea1208b-a0f4-4882-9da5-7e85331975b3","Type":"ContainerDied","Data":"f3f125bd4dd12a603f9f3e5dc167e506683fc2a55538910f3e47a2e1f1fa97c9"}
Oct 30 19:42:13.002249 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:13.002166    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3f125bd4dd12a603f9f3e5dc167e506683fc2a55538910f3e47a2e1f1fa97c9"
Oct 30 19:42:18.622737 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.622701    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"]
Oct 30 19:42:18.623189 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.623112    6030 cpu_manager.go:414] "containerMap: removing container" podUID="aea1208b-a0f4-4882-9da5-7e85331975b3" containerName="revoke-keys"
Oct 30 19:42:18.623189 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.623123    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea1208b-a0f4-4882-9da5-7e85331975b3" containerName="revoke-keys"
Oct 30 19:42:18.623189 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.623175    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea1208b-a0f4-4882-9da5-7e85331975b3" containerName="revoke-keys"
Oct 30 19:42:18.626433 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.626413    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"
Oct 30 19:42:18.635390 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.635365    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:18.635660 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.635371    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:42:18.635957 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.635748    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:42:18.636123 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.635832    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 19:42:18.636674 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.636652    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"]
Oct 30 19:42:18.810791 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.810758    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b1376e-f1d9-4d02-8371-cefc8896a416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebcddb67-qm6rs\" (UID: \"a8b1376e-f1d9-4d02-8371-cefc8896a416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"
Oct 30 19:42:18.810994 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.810969    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2k5c\" (UniqueName: \"kubernetes.io/projected/a8b1376e-f1d9-4d02-8371-cefc8896a416-kube-api-access-m2k5c\") pod \"maas-api-revoke-sub-ebcddb67-qm6rs\" (UID: \"a8b1376e-f1d9-4d02-8371-cefc8896a416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"
Oct 30 19:42:18.811106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.811033    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b1376e-f1d9-4d02-8371-cefc8896a416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebcddb67-qm6rs\" (UID: \"a8b1376e-f1d9-4d02-8371-cefc8896a416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"
Oct 30 19:42:18.911550 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.911515    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b1376e-f1d9-4d02-8371-cefc8896a416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebcddb67-qm6rs\" (UID: \"a8b1376e-f1d9-4d02-8371-cefc8896a416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"
Oct 30 19:42:18.911723 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.911562    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b1376e-f1d9-4d02-8371-cefc8896a416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebcddb67-qm6rs\" (UID: \"a8b1376e-f1d9-4d02-8371-cefc8896a416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"
Oct 30 19:42:18.911723 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.911644    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2k5c\" (UniqueName: \"kubernetes.io/projected/a8b1376e-f1d9-4d02-8371-cefc8896a416-kube-api-access-m2k5c\") pod \"maas-api-revoke-sub-ebcddb67-qm6rs\" (UID: \"a8b1376e-f1d9-4d02-8371-cefc8896a416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"
Oct 30 19:42:18.912312 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.912285    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b1376e-f1d9-4d02-8371-cefc8896a416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebcddb67-qm6rs\" (UID: \"a8b1376e-f1d9-4d02-8371-cefc8896a416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"
Oct 30 19:42:18.914037 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.914011    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b1376e-f1d9-4d02-8371-cefc8896a416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebcddb67-qm6rs\" (UID: \"a8b1376e-f1d9-4d02-8371-cefc8896a416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"
Oct 30 19:42:18.925684 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.925660    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2k5c\" (UniqueName: \"kubernetes.io/projected/a8b1376e-f1d9-4d02-8371-cefc8896a416-kube-api-access-m2k5c\") pod \"maas-api-revoke-sub-ebcddb67-qm6rs\" (UID: \"a8b1376e-f1d9-4d02-8371-cefc8896a416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"
Oct 30 19:42:18.940699 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:18.940678    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"
Oct 30 19:42:19.217481 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:19.217457    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"]
Oct 30 19:42:19.219913 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:42:19.219885    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b1376e_f1d9_4d02_8371_cefc8896a416.slice/crio-e14414805e119771cc6a71db44485290c41b330b5441d53090fe1ab7e3d68276 WatchSource:0}: Error finding container e14414805e119771cc6a71db44485290c41b330b5441d53090fe1ab7e3d68276: Status 404 returned error can't find the container with id e14414805e119771cc6a71db44485290c41b330b5441d53090fe1ab7e3d68276
Oct 30 19:42:20.037441 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:20.037407    6030 generic.go:356] "Generic (PLEG): container finished" podID="a8b1376e-f1d9-4d02-8371-cefc8896a416" containerID="1613cf0816eecbae0f44da51d8ec11ecc144dcce031d474aa8e291e27f577716" exitCode=0
Oct 30 19:42:20.037865 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:20.037488    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs" event={"ID":"a8b1376e-f1d9-4d02-8371-cefc8896a416","Type":"ContainerDied","Data":"1613cf0816eecbae0f44da51d8ec11ecc144dcce031d474aa8e291e27f577716"}
Oct 30 19:42:20.037865 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:20.037544    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs" event={"ID":"a8b1376e-f1d9-4d02-8371-cefc8896a416","Type":"ContainerStarted","Data":"e14414805e119771cc6a71db44485290c41b330b5441d53090fe1ab7e3d68276"}
Oct 30 19:42:21.178923 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:21.178894    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"
Oct 30 19:42:21.339107 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:21.339004    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b1376e-f1d9-4d02-8371-cefc8896a416-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b1376e-f1d9-4d02-8371-cefc8896a416-maas-api-xks-ca\") pod \"a8b1376e-f1d9-4d02-8371-cefc8896a416\" (UID: \"a8b1376e-f1d9-4d02-8371-cefc8896a416\") "
Oct 30 19:42:21.339107 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:21.339100    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b1376e-f1d9-4d02-8371-cefc8896a416-kube-api-access-m2k5c\" (UniqueName: \"kubernetes.io/projected/a8b1376e-f1d9-4d02-8371-cefc8896a416-kube-api-access-m2k5c\") pod \"a8b1376e-f1d9-4d02-8371-cefc8896a416\" (UID: \"a8b1376e-f1d9-4d02-8371-cefc8896a416\") "
Oct 30 19:42:21.339300 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:21.339186    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b1376e-f1d9-4d02-8371-cefc8896a416-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b1376e-f1d9-4d02-8371-cefc8896a416-maas-api-openshift-ca\") pod \"a8b1376e-f1d9-4d02-8371-cefc8896a416\" (UID: \"a8b1376e-f1d9-4d02-8371-cefc8896a416\") "
Oct 30 19:42:21.339592 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:21.339560    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b1376e-f1d9-4d02-8371-cefc8896a416-maas-api-openshift-ca" pod "a8b1376e-f1d9-4d02-8371-cefc8896a416" (UID: "a8b1376e-f1d9-4d02-8371-cefc8896a416"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:21.341252 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:21.341229    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b1376e-f1d9-4d02-8371-cefc8896a416-maas-api-xks-ca" pod "a8b1376e-f1d9-4d02-8371-cefc8896a416" (UID: "a8b1376e-f1d9-4d02-8371-cefc8896a416"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:21.341335 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:21.341316    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b1376e-f1d9-4d02-8371-cefc8896a416-kube-api-access-m2k5c" pod "a8b1376e-f1d9-4d02-8371-cefc8896a416" (UID: "a8b1376e-f1d9-4d02-8371-cefc8896a416"). InnerVolumeSpecName "kube-api-access-m2k5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:21.440067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:21.440015    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b1376e-f1d9-4d02-8371-cefc8896a416-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:21.440067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:21.440038    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2k5c\" (UniqueName: \"kubernetes.io/projected/a8b1376e-f1d9-4d02-8371-cefc8896a416-kube-api-access-m2k5c\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:21.440271 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:21.440084    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b1376e-f1d9-4d02-8371-cefc8896a416-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:22.049060 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:22.049000    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs" event={"ID":"a8b1376e-f1d9-4d02-8371-cefc8896a416","Type":"ContainerDied","Data":"e14414805e119771cc6a71db44485290c41b330b5441d53090fe1ab7e3d68276"}
Oct 30 19:42:22.049274 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:22.049035    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"
Oct 30 19:42:22.049274 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:22.049110    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e14414805e119771cc6a71db44485290c41b330b5441d53090fe1ab7e3d68276"
Oct 30 19:42:26.349347 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.349312    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"]
Oct 30 19:42:26.349717 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.349699    6030 cpu_manager.go:414] "containerMap: removing container" podUID="a8b1376e-f1d9-4d02-8371-cefc8896a416" containerName="revoke-keys"
Oct 30 19:42:26.349717 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.349710    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b1376e-f1d9-4d02-8371-cefc8896a416" containerName="revoke-keys"
Oct 30 19:42:26.349810 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.349799    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b1376e-f1d9-4d02-8371-cefc8896a416" containerName="revoke-keys"
Oct 30 19:42:26.353223 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.353207    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.382061 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.382022    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"]
Oct 30 19:42:26.385361 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.385340    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-8836a8-openshift-default-dockercfg-dvhtk\""
Oct 30 19:42:26.399655 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.399626    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/346006b7-21cc-469c-b8db-f82278c28c99-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.399777 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.399664    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.399777 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.399682    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/346006b7-21cc-469c-b8db-f82278c28c99-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.399874 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.399851    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgfhz\" (UniqueName: \"kubernetes.io/projected/346006b7-21cc-469c-b8db-f82278c28c99-kube-api-access-sgfhz\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.399921 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.399891    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/346006b7-21cc-469c-b8db-f82278c28c99-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.399921 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.399913    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.399987 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.399975    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.400027 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.400016    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.400087 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.400036    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.501088 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.501031    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgfhz\" (UniqueName: \"kubernetes.io/projected/346006b7-21cc-469c-b8db-f82278c28c99-kube-api-access-sgfhz\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.501088 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.501080    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/346006b7-21cc-469c-b8db-f82278c28c99-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.501320 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.501102    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.501320 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.501119    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.501320 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.501150    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.501320 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.501168    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.501320 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.501252    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/346006b7-21cc-469c-b8db-f82278c28c99-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.501320 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.501319    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.501610 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.501343    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/346006b7-21cc-469c-b8db-f82278c28c99-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.501610 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.501549    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.501712 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.501613    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.501712 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.501666    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.501712 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.501685    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.502094 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.502019    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/346006b7-21cc-469c-b8db-f82278c28c99-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.503725 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.503700    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.503839 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.503781    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/346006b7-21cc-469c-b8db-f82278c28c99-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.515131 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.515100    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/346006b7-21cc-469c-b8db-f82278c28c99-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.515245 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.515204    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgfhz\" (UniqueName: \"kubernetes.io/projected/346006b7-21cc-469c-b8db-f82278c28c99-kube-api-access-sgfhz\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.666547 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.666512    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:26.973467 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.973373    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"]
Oct 30 19:42:26.981540 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:42:26.981509    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod346006b7_21cc_469c_b8db_f82278c28c99.slice/crio-e7adbd4d04f44fd2f101e0ef9987440d119d4d33da57be7fb6e9dbb7db675611 WatchSource:0}: Error finding container e7adbd4d04f44fd2f101e0ef9987440d119d4d33da57be7fb6e9dbb7db675611: Status 404 returned error can't find the container with id e7adbd4d04f44fd2f101e0ef9987440d119d4d33da57be7fb6e9dbb7db675611
Oct 30 19:42:26.983900 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.983854    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:42:26.983995 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.983949    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:42:26.983995 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:26.983988    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:42:27.075099 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:27.075067    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4" event={"ID":"346006b7-21cc-469c-b8db-f82278c28c99","Type":"ContainerStarted","Data":"a0178749bf7bf8543f8d30b9125c08f118ec049576388b53b5833f89b75c0e28"}
Oct 30 19:42:27.075275 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:27.075118    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4" event={"ID":"346006b7-21cc-469c-b8db-f82278c28c99","Type":"ContainerStarted","Data":"e7adbd4d04f44fd2f101e0ef9987440d119d4d33da57be7fb6e9dbb7db675611"}
Oct 30 19:42:27.111430 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:27.111380    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4" podStartSLOduration=1.111361332 podStartE2EDuration="1.111361332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:27.109174149 +0000 UTC m=+626.024781009" watchObservedRunningTime="2026-10-30 19:42:27.111361332 +0000 UTC m=+626.026968188"
Oct 30 19:42:27.667258 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:27.667219    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:28.668640 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:28.668606    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 19:42:28.669075 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:28.668671    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4" podUID="346006b7-21cc-469c-b8db-f82278c28c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 19:42:29.667627 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:29.667596    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 19:42:29.667816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:29.667648    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4" podUID="346006b7-21cc-469c-b8db-f82278c28c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 19:42:30.667832 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:30.667799    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 19:42:30.668263 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:30.667855    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4" podUID="346006b7-21cc-469c-b8db-f82278c28c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 19:42:31.667035 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:31.666995    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 19:42:31.667246 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:31.667067    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4" podUID="346006b7-21cc-469c-b8db-f82278c28c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 19:42:32.667553 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:32.667514    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 19:42:32.668037 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:32.667575    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4" podUID="346006b7-21cc-469c-b8db-f82278c28c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 19:42:33.324998 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.324964    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"]
Oct 30 19:42:33.328827 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.328805    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"
Oct 30 19:42:33.351178 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.351145    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"]
Oct 30 19:42:33.357784 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.357742    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:42:33.357784 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.357777    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:42:33.358021 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.358004    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:33.359572 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.359553    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 19:42:33.397095 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.397060    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4725\" (UniqueName: \"kubernetes.io/projected/76b58bf3-e970-4384-9f03-faacc5c95f4c-kube-api-access-k4725\") pod \"maas-api-revoke-sub-0e11f28b-dsfrf\" (UID: \"76b58bf3-e970-4384-9f03-faacc5c95f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"
Oct 30 19:42:33.397255 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.397190    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76b58bf3-e970-4384-9f03-faacc5c95f4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e11f28b-dsfrf\" (UID: \"76b58bf3-e970-4384-9f03-faacc5c95f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"
Oct 30 19:42:33.397255 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.397225    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76b58bf3-e970-4384-9f03-faacc5c95f4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e11f28b-dsfrf\" (UID: \"76b58bf3-e970-4384-9f03-faacc5c95f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"
Oct 30 19:42:33.498152 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.498119    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76b58bf3-e970-4384-9f03-faacc5c95f4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e11f28b-dsfrf\" (UID: \"76b58bf3-e970-4384-9f03-faacc5c95f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"
Oct 30 19:42:33.498152 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.498154    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76b58bf3-e970-4384-9f03-faacc5c95f4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e11f28b-dsfrf\" (UID: \"76b58bf3-e970-4384-9f03-faacc5c95f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"
Oct 30 19:42:33.498413 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.498219    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4725\" (UniqueName: \"kubernetes.io/projected/76b58bf3-e970-4384-9f03-faacc5c95f4c-kube-api-access-k4725\") pod \"maas-api-revoke-sub-0e11f28b-dsfrf\" (UID: \"76b58bf3-e970-4384-9f03-faacc5c95f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"
Oct 30 19:42:33.498895 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.498869    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76b58bf3-e970-4384-9f03-faacc5c95f4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e11f28b-dsfrf\" (UID: \"76b58bf3-e970-4384-9f03-faacc5c95f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"
Oct 30 19:42:33.500659 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.500635    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76b58bf3-e970-4384-9f03-faacc5c95f4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e11f28b-dsfrf\" (UID: \"76b58bf3-e970-4384-9f03-faacc5c95f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"
Oct 30 19:42:33.511711 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.511686    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4725\" (UniqueName: \"kubernetes.io/projected/76b58bf3-e970-4384-9f03-faacc5c95f4c-kube-api-access-k4725\") pod \"maas-api-revoke-sub-0e11f28b-dsfrf\" (UID: \"76b58bf3-e970-4384-9f03-faacc5c95f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"
Oct 30 19:42:33.642642 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.642599    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"
Oct 30 19:42:33.722920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.721346    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:33.722920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.722600    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:33.724243 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.723988    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:42:33.925173 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:33.925147    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"]
Oct 30 19:42:33.927553 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:42:33.927523    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b58bf3_e970_4384_9f03_faacc5c95f4c.slice/crio-fa004db382273cd21f86ae81df188c309a61380678d8c1ac90ba270b1dbd62c5 WatchSource:0}: Error finding container fa004db382273cd21f86ae81df188c309a61380678d8c1ac90ba270b1dbd62c5: Status 404 returned error can't find the container with id fa004db382273cd21f86ae81df188c309a61380678d8c1ac90ba270b1dbd62c5
Oct 30 19:42:34.127121 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:34.127093    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf" event={"ID":"76b58bf3-e970-4384-9f03-faacc5c95f4c","Type":"ContainerStarted","Data":"3ce89d4d2f60479326ea1031cdc5512f5ad7d05042a0777da497673f0fbbd698"}
Oct 30 19:42:34.127270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:34.127144    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf" event={"ID":"76b58bf3-e970-4384-9f03-faacc5c95f4c","Type":"ContainerStarted","Data":"fa004db382273cd21f86ae81df188c309a61380678d8c1ac90ba270b1dbd62c5"}
Oct 30 19:42:34.146683 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:34.146632    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf" podStartSLOduration=1.146615636 podStartE2EDuration="1.146615636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:34.146387147 +0000 UTC m=+633.061994003" watchObservedRunningTime="2026-10-30 19:42:34.146615636 +0000 UTC m=+633.062222507"
Oct 30 19:42:34.287429 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:42:34.287330    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:42:35.133606 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:35.133574    6030 generic.go:356] "Generic (PLEG): container finished" podID="76b58bf3-e970-4384-9f03-faacc5c95f4c" containerID="3ce89d4d2f60479326ea1031cdc5512f5ad7d05042a0777da497673f0fbbd698" exitCode=0
Oct 30 19:42:35.134169 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:35.133653    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf" event={"ID":"76b58bf3-e970-4384-9f03-faacc5c95f4c","Type":"ContainerDied","Data":"3ce89d4d2f60479326ea1031cdc5512f5ad7d05042a0777da497673f0fbbd698"}
Oct 30 19:42:36.277702 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:36.277679    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"
Oct 30 19:42:36.441639 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:36.441564    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b58bf3-e970-4384-9f03-faacc5c95f4c-kube-api-access-k4725\" (UniqueName: \"kubernetes.io/projected/76b58bf3-e970-4384-9f03-faacc5c95f4c-kube-api-access-k4725\") pod \"76b58bf3-e970-4384-9f03-faacc5c95f4c\" (UID: \"76b58bf3-e970-4384-9f03-faacc5c95f4c\") "
Oct 30 19:42:36.441790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:36.441684    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76b58bf3-e970-4384-9f03-faacc5c95f4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76b58bf3-e970-4384-9f03-faacc5c95f4c-maas-api-openshift-ca\") pod \"76b58bf3-e970-4384-9f03-faacc5c95f4c\" (UID: \"76b58bf3-e970-4384-9f03-faacc5c95f4c\") "
Oct 30 19:42:36.441790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:36.441701    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76b58bf3-e970-4384-9f03-faacc5c95f4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76b58bf3-e970-4384-9f03-faacc5c95f4c-maas-api-xks-ca\") pod \"76b58bf3-e970-4384-9f03-faacc5c95f4c\" (UID: \"76b58bf3-e970-4384-9f03-faacc5c95f4c\") "
Oct 30 19:42:36.442025 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:36.441996    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b58bf3-e970-4384-9f03-faacc5c95f4c-maas-api-openshift-ca" pod "76b58bf3-e970-4384-9f03-faacc5c95f4c" (UID: "76b58bf3-e970-4384-9f03-faacc5c95f4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:36.443926 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:36.443901    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b58bf3-e970-4384-9f03-faacc5c95f4c-maas-api-xks-ca" pod "76b58bf3-e970-4384-9f03-faacc5c95f4c" (UID: "76b58bf3-e970-4384-9f03-faacc5c95f4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:36.444010 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:36.443905    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b58bf3-e970-4384-9f03-faacc5c95f4c-kube-api-access-k4725" pod "76b58bf3-e970-4384-9f03-faacc5c95f4c" (UID: "76b58bf3-e970-4384-9f03-faacc5c95f4c"). InnerVolumeSpecName "kube-api-access-k4725". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:36.542615 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:36.542581    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4725\" (UniqueName: \"kubernetes.io/projected/76b58bf3-e970-4384-9f03-faacc5c95f4c-kube-api-access-k4725\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:36.542615 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:36.542611    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76b58bf3-e970-4384-9f03-faacc5c95f4c-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:36.542615 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:36.542621    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76b58bf3-e970-4384-9f03-faacc5c95f4c-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:37.146315 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:37.146287    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"
Oct 30 19:42:37.146505 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:37.146325    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf" event={"ID":"76b58bf3-e970-4384-9f03-faacc5c95f4c","Type":"ContainerDied","Data":"fa004db382273cd21f86ae81df188c309a61380678d8c1ac90ba270b1dbd62c5"}
Oct 30 19:42:37.146505 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:37.146372    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa004db382273cd21f86ae81df188c309a61380678d8c1ac90ba270b1dbd62c5"
Oct 30 19:42:38.807468 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:38.801128    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"]
Oct 30 19:42:38.807468 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:38.802177    6030 cpu_manager.go:414] "containerMap: removing container" podUID="76b58bf3-e970-4384-9f03-faacc5c95f4c" containerName="revoke-keys"
Oct 30 19:42:38.807468 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:38.802207    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b58bf3-e970-4384-9f03-faacc5c95f4c" containerName="revoke-keys"
Oct 30 19:42:38.807468 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:38.802467    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b58bf3-e970-4384-9f03-faacc5c95f4c" containerName="revoke-keys"
Oct 30 19:42:38.808855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:38.808827    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"
Oct 30 19:42:38.811868 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:38.811823    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"]
Oct 30 19:42:38.813852 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:38.813823    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:38.813958 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:38.813849    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:42:38.813958 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:38.813936    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:42:38.814096 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:38.814082    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 19:42:38.972362 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:38.972329    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/357ea7bb-a182-4b5e-8f73-25b23e9370e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49cf390e-9cf7z\" (UID: \"357ea7bb-a182-4b5e-8f73-25b23e9370e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"
Oct 30 19:42:38.972543 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:38.972376    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/357ea7bb-a182-4b5e-8f73-25b23e9370e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49cf390e-9cf7z\" (UID: \"357ea7bb-a182-4b5e-8f73-25b23e9370e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"
Oct 30 19:42:38.972543 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:38.972506    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qxfl\" (UniqueName: \"kubernetes.io/projected/357ea7bb-a182-4b5e-8f73-25b23e9370e9-kube-api-access-9qxfl\") pod \"maas-api-revoke-sub-49cf390e-9cf7z\" (UID: \"357ea7bb-a182-4b5e-8f73-25b23e9370e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"
Oct 30 19:42:39.073802 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:39.073702    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/357ea7bb-a182-4b5e-8f73-25b23e9370e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49cf390e-9cf7z\" (UID: \"357ea7bb-a182-4b5e-8f73-25b23e9370e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"
Oct 30 19:42:39.073802 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:39.073796    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qxfl\" (UniqueName: \"kubernetes.io/projected/357ea7bb-a182-4b5e-8f73-25b23e9370e9-kube-api-access-9qxfl\") pod \"maas-api-revoke-sub-49cf390e-9cf7z\" (UID: \"357ea7bb-a182-4b5e-8f73-25b23e9370e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"
Oct 30 19:42:39.074070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:39.073850    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/357ea7bb-a182-4b5e-8f73-25b23e9370e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49cf390e-9cf7z\" (UID: \"357ea7bb-a182-4b5e-8f73-25b23e9370e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"
Oct 30 19:42:39.074532 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:39.074503    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/357ea7bb-a182-4b5e-8f73-25b23e9370e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49cf390e-9cf7z\" (UID: \"357ea7bb-a182-4b5e-8f73-25b23e9370e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"
Oct 30 19:42:39.076561 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:39.076540    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/357ea7bb-a182-4b5e-8f73-25b23e9370e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49cf390e-9cf7z\" (UID: \"357ea7bb-a182-4b5e-8f73-25b23e9370e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"
Oct 30 19:42:39.087857 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:39.087834    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qxfl\" (UniqueName: \"kubernetes.io/projected/357ea7bb-a182-4b5e-8f73-25b23e9370e9-kube-api-access-9qxfl\") pod \"maas-api-revoke-sub-49cf390e-9cf7z\" (UID: \"357ea7bb-a182-4b5e-8f73-25b23e9370e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"
Oct 30 19:42:39.124853 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:39.124826    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"
Oct 30 19:42:39.429821 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:39.429799    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"]
Oct 30 19:42:39.431332 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:42:39.431303    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357ea7bb_a182_4b5e_8f73_25b23e9370e9.slice/crio-8fdb94c9469d4378ce940fc911a0462447e5b0810e6baf1624a7a2031aa3ca41 WatchSource:0}: Error finding container 8fdb94c9469d4378ce940fc911a0462447e5b0810e6baf1624a7a2031aa3ca41: Status 404 returned error can't find the container with id 8fdb94c9469d4378ce940fc911a0462447e5b0810e6baf1624a7a2031aa3ca41
Oct 30 19:42:40.162651 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:40.162616    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z" event={"ID":"357ea7bb-a182-4b5e-8f73-25b23e9370e9","Type":"ContainerStarted","Data":"aea62fc534e99a7284ba7235e84c9c0c48c96928d0f33c5bb5a960ff8d7def9e"}
Oct 30 19:42:40.163114 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:40.162664    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z" event={"ID":"357ea7bb-a182-4b5e-8f73-25b23e9370e9","Type":"ContainerStarted","Data":"8fdb94c9469d4378ce940fc911a0462447e5b0810e6baf1624a7a2031aa3ca41"}
Oct 30 19:42:40.197403 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:40.193276    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z" podStartSLOduration=2.1932529450000002 podStartE2EDuration="2.193252945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:40.19049015 +0000 UTC m=+639.106097008" watchObservedRunningTime="2026-10-30 19:42:40.193252945 +0000 UTC m=+639.108859796"
Oct 30 19:42:41.169113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:41.169010    6030 generic.go:356] "Generic (PLEG): container finished" podID="357ea7bb-a182-4b5e-8f73-25b23e9370e9" containerID="aea62fc534e99a7284ba7235e84c9c0c48c96928d0f33c5bb5a960ff8d7def9e" exitCode=0
Oct 30 19:42:41.169113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:41.169087    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z" event={"ID":"357ea7bb-a182-4b5e-8f73-25b23e9370e9","Type":"ContainerDied","Data":"aea62fc534e99a7284ba7235e84c9c0c48c96928d0f33c5bb5a960ff8d7def9e"}
Oct 30 19:42:42.312335 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:42.312309    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"
Oct 30 19:42:42.424577 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:42.424485    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357ea7bb-a182-4b5e-8f73-25b23e9370e9-kube-api-access-9qxfl\" (UniqueName: \"kubernetes.io/projected/357ea7bb-a182-4b5e-8f73-25b23e9370e9-kube-api-access-9qxfl\") pod \"357ea7bb-a182-4b5e-8f73-25b23e9370e9\" (UID: \"357ea7bb-a182-4b5e-8f73-25b23e9370e9\") "
Oct 30 19:42:42.424730 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:42.424603    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/357ea7bb-a182-4b5e-8f73-25b23e9370e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/357ea7bb-a182-4b5e-8f73-25b23e9370e9-maas-api-xks-ca\") pod \"357ea7bb-a182-4b5e-8f73-25b23e9370e9\" (UID: \"357ea7bb-a182-4b5e-8f73-25b23e9370e9\") "
Oct 30 19:42:42.424730 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:42.424645    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/357ea7bb-a182-4b5e-8f73-25b23e9370e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/357ea7bb-a182-4b5e-8f73-25b23e9370e9-maas-api-openshift-ca\") pod \"357ea7bb-a182-4b5e-8f73-25b23e9370e9\" (UID: \"357ea7bb-a182-4b5e-8f73-25b23e9370e9\") "
Oct 30 19:42:42.425097 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:42.425035    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357ea7bb-a182-4b5e-8f73-25b23e9370e9-maas-api-openshift-ca" pod "357ea7bb-a182-4b5e-8f73-25b23e9370e9" (UID: "357ea7bb-a182-4b5e-8f73-25b23e9370e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:42.426909 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:42.426881    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357ea7bb-a182-4b5e-8f73-25b23e9370e9-kube-api-access-9qxfl" pod "357ea7bb-a182-4b5e-8f73-25b23e9370e9" (UID: "357ea7bb-a182-4b5e-8f73-25b23e9370e9"). InnerVolumeSpecName "kube-api-access-9qxfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:42.427028 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:42.426924    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357ea7bb-a182-4b5e-8f73-25b23e9370e9-maas-api-xks-ca" pod "357ea7bb-a182-4b5e-8f73-25b23e9370e9" (UID: "357ea7bb-a182-4b5e-8f73-25b23e9370e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:42.525730 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:42.525694    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qxfl\" (UniqueName: \"kubernetes.io/projected/357ea7bb-a182-4b5e-8f73-25b23e9370e9-kube-api-access-9qxfl\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:42.525730 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:42.525724    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/357ea7bb-a182-4b5e-8f73-25b23e9370e9-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:42.525730 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:42.525737    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/357ea7bb-a182-4b5e-8f73-25b23e9370e9-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:43.179576 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:43.179541    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z" event={"ID":"357ea7bb-a182-4b5e-8f73-25b23e9370e9","Type":"ContainerDied","Data":"8fdb94c9469d4378ce940fc911a0462447e5b0810e6baf1624a7a2031aa3ca41"}
Oct 30 19:42:43.179751 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:43.179589    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fdb94c9469d4378ce940fc911a0462447e5b0810e6baf1624a7a2031aa3ca41"
Oct 30 19:42:43.179751 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:43.179549    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"
Oct 30 19:42:46.167758 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.167725    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"]
Oct 30 19:42:46.168207 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.168191    6030 cpu_manager.go:414] "containerMap: removing container" podUID="357ea7bb-a182-4b5e-8f73-25b23e9370e9" containerName="revoke-keys"
Oct 30 19:42:46.168263 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.168208    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357ea7bb-a182-4b5e-8f73-25b23e9370e9" containerName="revoke-keys"
Oct 30 19:42:46.168298 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.168285    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="357ea7bb-a182-4b5e-8f73-25b23e9370e9" containerName="revoke-keys"
Oct 30 19:42:46.171583 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.171562    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.193921 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.193889    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"]
Oct 30 19:42:46.194324 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.194304    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-c1fdd6-openshift-default-dockercfg-7mt7d\""
Oct 30 19:42:46.367394 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.367356    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.367582 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.367399    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.367582 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.367420    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.367582 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.367474    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.367582 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.367542    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.367582 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.367568    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.367768 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.367742    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjxv6\" (UniqueName: \"kubernetes.io/projected/be081d10-b3c7-4a5a-9abf-9989ca04e364-kube-api-access-zjxv6\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.367844 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.367827    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be081d10-b3c7-4a5a-9abf-9989ca04e364-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.367961 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.367940    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.468596 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.468502    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.468596 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.468547    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.468848 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.468664    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.468848 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.468763    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.468848 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.468797    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.468848 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.468846    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjxv6\" (UniqueName: \"kubernetes.io/projected/be081d10-b3c7-4a5a-9abf-9989ca04e364-kube-api-access-zjxv6\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.469112 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.468881    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be081d10-b3c7-4a5a-9abf-9989ca04e364-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.469112 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.468933    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.469112 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.468941    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.469112 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.468983    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.469112 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.469016    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.469318 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.469264    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.469356 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.469331    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.469891 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.469859    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be081d10-b3c7-4a5a-9abf-9989ca04e364-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.471278 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.471256    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.471476 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.471459    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.485569 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.485537    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.485654 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.485630    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjxv6\" (UniqueName: \"kubernetes.io/projected/be081d10-b3c7-4a5a-9abf-9989ca04e364-kube-api-access-zjxv6\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:46.784669 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:46.784559    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:47.080691 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:42:47.080664    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe081d10_b3c7_4a5a_9abf_9989ca04e364.slice/crio-74c23fdb7d941a5f59023aa2250b9e176760f15a9dfffab64d21d7dd923b582e WatchSource:0}: Error finding container 74c23fdb7d941a5f59023aa2250b9e176760f15a9dfffab64d21d7dd923b582e: Status 404 returned error can't find the container with id 74c23fdb7d941a5f59023aa2250b9e176760f15a9dfffab64d21d7dd923b582e
Oct 30 19:42:47.082038 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.082009    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"]
Oct 30 19:42:47.083305 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.083255    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:42:47.083412 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.083342    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:42:47.083474 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.083415    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:42:47.202727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.202674    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5" event={"ID":"be081d10-b3c7-4a5a-9abf-9989ca04e364","Type":"ContainerStarted","Data":"777297e7b845203657f6165e0e2eed83a5e4b838f0246bf2b2a02e687a5e2ae5"}
Oct 30 19:42:47.203247 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.202742    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5" event={"ID":"be081d10-b3c7-4a5a-9abf-9989ca04e364","Type":"ContainerStarted","Data":"74c23fdb7d941a5f59023aa2250b9e176760f15a9dfffab64d21d7dd923b582e"}
Oct 30 19:42:47.364933 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.364824    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5" podStartSLOduration=1.364810192 podStartE2EDuration="1.364810192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:47.241797346 +0000 UTC m=+646.157404203" watchObservedRunningTime="2026-10-30 19:42:47.364810192 +0000 UTC m=+646.280417048"
Oct 30 19:42:47.365143 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.365019    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"]
Oct 30 19:42:47.368732 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.368707    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"
Oct 30 19:42:47.386092 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.376122    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 19:42:47.386092 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.376738    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:47.386092 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.376829    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:42:47.386092 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.376887    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:42:47.393829 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.393319    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"]
Oct 30 19:42:47.495618 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.495575    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89527289-dff4-4c7e-9eeb-6e981faf1a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96878bba-g5xwq\" (UID: \"89527289-dff4-4c7e-9eeb-6e981faf1a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"
Oct 30 19:42:47.495802 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.495673    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9w2j\" (UniqueName: \"kubernetes.io/projected/89527289-dff4-4c7e-9eeb-6e981faf1a21-kube-api-access-d9w2j\") pod \"maas-api-revoke-sub-96878bba-g5xwq\" (UID: \"89527289-dff4-4c7e-9eeb-6e981faf1a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"
Oct 30 19:42:47.495802 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.495755    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89527289-dff4-4c7e-9eeb-6e981faf1a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96878bba-g5xwq\" (UID: \"89527289-dff4-4c7e-9eeb-6e981faf1a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"
Oct 30 19:42:47.597331 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.597291    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89527289-dff4-4c7e-9eeb-6e981faf1a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96878bba-g5xwq\" (UID: \"89527289-dff4-4c7e-9eeb-6e981faf1a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"
Oct 30 19:42:47.597494 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.597386    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89527289-dff4-4c7e-9eeb-6e981faf1a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96878bba-g5xwq\" (UID: \"89527289-dff4-4c7e-9eeb-6e981faf1a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"
Oct 30 19:42:47.597494 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.597467    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9w2j\" (UniqueName: \"kubernetes.io/projected/89527289-dff4-4c7e-9eeb-6e981faf1a21-kube-api-access-d9w2j\") pod \"maas-api-revoke-sub-96878bba-g5xwq\" (UID: \"89527289-dff4-4c7e-9eeb-6e981faf1a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"
Oct 30 19:42:47.598201 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.598156    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89527289-dff4-4c7e-9eeb-6e981faf1a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96878bba-g5xwq\" (UID: \"89527289-dff4-4c7e-9eeb-6e981faf1a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"
Oct 30 19:42:47.600115 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.600089    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89527289-dff4-4c7e-9eeb-6e981faf1a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96878bba-g5xwq\" (UID: \"89527289-dff4-4c7e-9eeb-6e981faf1a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"
Oct 30 19:42:47.611132 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.611108    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9w2j\" (UniqueName: \"kubernetes.io/projected/89527289-dff4-4c7e-9eeb-6e981faf1a21-kube-api-access-d9w2j\") pod \"maas-api-revoke-sub-96878bba-g5xwq\" (UID: \"89527289-dff4-4c7e-9eeb-6e981faf1a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"
Oct 30 19:42:47.715301 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.715270    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"
Oct 30 19:42:47.810232 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:47.786239    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:48.013904 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:48.013828    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"]
Oct 30 19:42:48.049069 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:42:48.045253    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89527289_dff4_4c7e_9eeb_6e981faf1a21.slice/crio-f1448175d848a82e659e6a838a3ff01348a998927ae40629817c4d3eba611179 WatchSource:0}: Error finding container f1448175d848a82e659e6a838a3ff01348a998927ae40629817c4d3eba611179: Status 404 returned error can't find the container with id f1448175d848a82e659e6a838a3ff01348a998927ae40629817c4d3eba611179
Oct 30 19:42:48.221667 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:48.221360    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq" event={"ID":"89527289-dff4-4c7e-9eeb-6e981faf1a21","Type":"ContainerStarted","Data":"f1448175d848a82e659e6a838a3ff01348a998927ae40629817c4d3eba611179"}
Oct 30 19:42:48.791076 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:48.790968    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 19:42:48.792101 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:48.791034    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5" podUID="be081d10-b3c7-4a5a-9abf-9989ca04e364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 19:42:49.250846 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:49.249667    6030 generic.go:356] "Generic (PLEG): container finished" podID="89527289-dff4-4c7e-9eeb-6e981faf1a21" containerID="436a508b3c22cfafc879bfb49b6c60d4e82e7c43d1bc127fb40d1ffea6012b7c" exitCode=0
Oct 30 19:42:49.250846 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:49.250707    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq" event={"ID":"89527289-dff4-4c7e-9eeb-6e981faf1a21","Type":"ContainerDied","Data":"436a508b3c22cfafc879bfb49b6c60d4e82e7c43d1bc127fb40d1ffea6012b7c"}
Oct 30 19:42:49.787910 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:49.787447    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 19:42:49.787910 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:49.787509    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5" podUID="be081d10-b3c7-4a5a-9abf-9989ca04e364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 19:42:50.797070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:50.792510    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 19:42:50.797070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:50.792583    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5" podUID="be081d10-b3c7-4a5a-9abf-9989ca04e364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 19:42:51.015073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.009066    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"
Oct 30 19:42:51.132076 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.127441    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89527289-dff4-4c7e-9eeb-6e981faf1a21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89527289-dff4-4c7e-9eeb-6e981faf1a21-maas-api-xks-ca\") pod \"89527289-dff4-4c7e-9eeb-6e981faf1a21\" (UID: \"89527289-dff4-4c7e-9eeb-6e981faf1a21\") "
Oct 30 19:42:51.140319 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.140285    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89527289-dff4-4c7e-9eeb-6e981faf1a21-kube-api-access-d9w2j\" (UniqueName: \"kubernetes.io/projected/89527289-dff4-4c7e-9eeb-6e981faf1a21-kube-api-access-d9w2j\") pod \"89527289-dff4-4c7e-9eeb-6e981faf1a21\" (UID: \"89527289-dff4-4c7e-9eeb-6e981faf1a21\") "
Oct 30 19:42:51.145079 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.140568    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89527289-dff4-4c7e-9eeb-6e981faf1a21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89527289-dff4-4c7e-9eeb-6e981faf1a21-maas-api-openshift-ca\") pod \"89527289-dff4-4c7e-9eeb-6e981faf1a21\" (UID: \"89527289-dff4-4c7e-9eeb-6e981faf1a21\") "
Oct 30 19:42:51.161288 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.161175    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89527289-dff4-4c7e-9eeb-6e981faf1a21-maas-api-xks-ca" pod "89527289-dff4-4c7e-9eeb-6e981faf1a21" (UID: "89527289-dff4-4c7e-9eeb-6e981faf1a21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:51.165684 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.161582    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89527289-dff4-4c7e-9eeb-6e981faf1a21-maas-api-openshift-ca" pod "89527289-dff4-4c7e-9eeb-6e981faf1a21" (UID: "89527289-dff4-4c7e-9eeb-6e981faf1a21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:51.284076 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.250441    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89527289-dff4-4c7e-9eeb-6e981faf1a21-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:51.284331 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.284308    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89527289-dff4-4c7e-9eeb-6e981faf1a21-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:51.284430 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.260071    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89527289-dff4-4c7e-9eeb-6e981faf1a21-kube-api-access-d9w2j" pod "89527289-dff4-4c7e-9eeb-6e981faf1a21" (UID: "89527289-dff4-4c7e-9eeb-6e981faf1a21"). InnerVolumeSpecName "kube-api-access-d9w2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:51.334072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.333354    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq" event={"ID":"89527289-dff4-4c7e-9eeb-6e981faf1a21","Type":"ContainerDied","Data":"f1448175d848a82e659e6a838a3ff01348a998927ae40629817c4d3eba611179"}
Oct 30 19:42:51.334072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.333418    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1448175d848a82e659e6a838a3ff01348a998927ae40629817c4d3eba611179"
Oct 30 19:42:51.334072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.333493    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"
Oct 30 19:42:51.387956 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.387002    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9w2j\" (UniqueName: \"kubernetes.io/projected/89527289-dff4-4c7e-9eeb-6e981faf1a21-kube-api-access-d9w2j\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:51.605167 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.600304    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"]
Oct 30 19:42:51.605167 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.600964    6030 cpu_manager.go:414] "containerMap: removing container" podUID="89527289-dff4-4c7e-9eeb-6e981faf1a21" containerName="revoke-keys"
Oct 30 19:42:51.605167 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.600984    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89527289-dff4-4c7e-9eeb-6e981faf1a21" containerName="revoke-keys"
Oct 30 19:42:51.614181 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.614147    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="89527289-dff4-4c7e-9eeb-6e981faf1a21" containerName="revoke-keys"
Oct 30 19:42:51.618112 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.618085    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"
Oct 30 19:42:51.628460 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.628432    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:51.628990 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.628968    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:42:51.629308 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.629291    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 19:42:51.630122 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.630103    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:42:51.636021 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.635900    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"]
Oct 30 19:42:51.786445 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.786404    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 19:42:51.786798 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.786764    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5" podUID="be081d10-b3c7-4a5a-9abf-9989ca04e364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 19:42:51.802108 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.797449    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hz74\" (UniqueName: \"kubernetes.io/projected/0f697726-1e6f-4463-b00f-0f4090608380-kube-api-access-2hz74\") pod \"maas-api-revoke-sub-a7b31cf5-t6xfr\" (UID: \"0f697726-1e6f-4463-b00f-0f4090608380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"
Oct 30 19:42:51.802790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.802761    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f697726-1e6f-4463-b00f-0f4090608380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7b31cf5-t6xfr\" (UID: \"0f697726-1e6f-4463-b00f-0f4090608380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"
Oct 30 19:42:51.803037 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.803018    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f697726-1e6f-4463-b00f-0f4090608380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7b31cf5-t6xfr\" (UID: \"0f697726-1e6f-4463-b00f-0f4090608380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"
Oct 30 19:42:51.903910 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.903879    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f697726-1e6f-4463-b00f-0f4090608380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7b31cf5-t6xfr\" (UID: \"0f697726-1e6f-4463-b00f-0f4090608380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"
Oct 30 19:42:51.904093 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.903977    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hz74\" (UniqueName: \"kubernetes.io/projected/0f697726-1e6f-4463-b00f-0f4090608380-kube-api-access-2hz74\") pod \"maas-api-revoke-sub-a7b31cf5-t6xfr\" (UID: \"0f697726-1e6f-4463-b00f-0f4090608380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"
Oct 30 19:42:51.904093 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.904015    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f697726-1e6f-4463-b00f-0f4090608380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7b31cf5-t6xfr\" (UID: \"0f697726-1e6f-4463-b00f-0f4090608380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"
Oct 30 19:42:51.906088 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.906059    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f697726-1e6f-4463-b00f-0f4090608380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7b31cf5-t6xfr\" (UID: \"0f697726-1e6f-4463-b00f-0f4090608380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"
Oct 30 19:42:51.906858 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.906836    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f697726-1e6f-4463-b00f-0f4090608380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7b31cf5-t6xfr\" (UID: \"0f697726-1e6f-4463-b00f-0f4090608380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"
Oct 30 19:42:51.919976 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.919838    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hz74\" (UniqueName: \"kubernetes.io/projected/0f697726-1e6f-4463-b00f-0f4090608380-kube-api-access-2hz74\") pod \"maas-api-revoke-sub-a7b31cf5-t6xfr\" (UID: \"0f697726-1e6f-4463-b00f-0f4090608380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"
Oct 30 19:42:51.956579 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:51.956553    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"
Oct 30 19:42:52.295077 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:52.294771    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"]
Oct 30 19:42:52.340423 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:52.340394    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr" event={"ID":"0f697726-1e6f-4463-b00f-0f4090608380","Type":"ContainerStarted","Data":"cf25d1ee461641eeeac6ade05cd2c701641bac60185466cb195fe2e76f3dd697"}
Oct 30 19:42:52.798066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:52.795441    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 19:42:52.798066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:52.795502    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5" podUID="be081d10-b3c7-4a5a-9abf-9989ca04e364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 19:42:53.360083 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:53.356841    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr" event={"ID":"0f697726-1e6f-4463-b00f-0f4090608380","Type":"ContainerStarted","Data":"2c3ce081b4d10e685e209b07a6992cbd1f07f189dbdc6240396c31f8b23de433"}
Oct 30 19:42:53.379892 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:53.379785    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr" podStartSLOduration=2.379770458 podStartE2EDuration="2.379770458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:53.379607558 +0000 UTC m=+652.295214416" watchObservedRunningTime="2026-10-30 19:42:53.379770458 +0000 UTC m=+652.295377315"
Oct 30 19:42:53.785297 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:53.785248    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 19:42:53.785513 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:53.785312    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5" podUID="be081d10-b3c7-4a5a-9abf-9989ca04e364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 19:42:54.380935 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:54.380404    6030 generic.go:356] "Generic (PLEG): container finished" podID="0f697726-1e6f-4463-b00f-0f4090608380" containerID="2c3ce081b4d10e685e209b07a6992cbd1f07f189dbdc6240396c31f8b23de433" exitCode=0
Oct 30 19:42:54.380935 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:54.380563    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr" event={"ID":"0f697726-1e6f-4463-b00f-0f4090608380","Type":"ContainerDied","Data":"2c3ce081b4d10e685e209b07a6992cbd1f07f189dbdc6240396c31f8b23de433"}
Oct 30 19:42:55.754969 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:55.753475    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:55.754969 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:55.754919    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:55.764092 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:55.756037    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 19:42:55.837181 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:55.832364    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"
Oct 30 19:42:55.959744 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:55.959655    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f697726-1e6f-4463-b00f-0f4090608380-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f697726-1e6f-4463-b00f-0f4090608380-maas-api-openshift-ca\") pod \"0f697726-1e6f-4463-b00f-0f4090608380\" (UID: \"0f697726-1e6f-4463-b00f-0f4090608380\") "
Oct 30 19:42:55.959744 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:55.959728    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f697726-1e6f-4463-b00f-0f4090608380-kube-api-access-2hz74\" (UniqueName: \"kubernetes.io/projected/0f697726-1e6f-4463-b00f-0f4090608380-kube-api-access-2hz74\") pod \"0f697726-1e6f-4463-b00f-0f4090608380\" (UID: \"0f697726-1e6f-4463-b00f-0f4090608380\") "
Oct 30 19:42:55.959970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:55.959798    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f697726-1e6f-4463-b00f-0f4090608380-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f697726-1e6f-4463-b00f-0f4090608380-maas-api-xks-ca\") pod \"0f697726-1e6f-4463-b00f-0f4090608380\" (UID: \"0f697726-1e6f-4463-b00f-0f4090608380\") "
Oct 30 19:42:55.960969 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:55.960940    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f697726-1e6f-4463-b00f-0f4090608380-maas-api-openshift-ca" pod "0f697726-1e6f-4463-b00f-0f4090608380" (UID: "0f697726-1e6f-4463-b00f-0f4090608380"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:55.964824 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:55.964794    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f697726-1e6f-4463-b00f-0f4090608380-maas-api-xks-ca" pod "0f697726-1e6f-4463-b00f-0f4090608380" (UID: "0f697726-1e6f-4463-b00f-0f4090608380"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:55.965435 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:55.965405    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f697726-1e6f-4463-b00f-0f4090608380-kube-api-access-2hz74" pod "0f697726-1e6f-4463-b00f-0f4090608380" (UID: "0f697726-1e6f-4463-b00f-0f4090608380"). InnerVolumeSpecName "kube-api-access-2hz74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:56.060973 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:56.060941    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f697726-1e6f-4463-b00f-0f4090608380-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:56.060973 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:56.060968    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hz74\" (UniqueName: \"kubernetes.io/projected/0f697726-1e6f-4463-b00f-0f4090608380-kube-api-access-2hz74\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:56.060973 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:56.060978    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f697726-1e6f-4463-b00f-0f4090608380-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:56.398350 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:56.398321    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"
Oct 30 19:42:56.398502 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:56.398352    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr" event={"ID":"0f697726-1e6f-4463-b00f-0f4090608380","Type":"ContainerDied","Data":"cf25d1ee461641eeeac6ade05cd2c701641bac60185466cb195fe2e76f3dd697"}
Oct 30 19:42:56.398502 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:42:56.398407    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf25d1ee461641eeeac6ade05cd2c701641bac60185466cb195fe2e76f3dd697"
Oct 30 19:43:01.857422 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:01.857389    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"]
Oct 30 19:43:01.857918 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:01.857812    6030 cpu_manager.go:414] "containerMap: removing container" podUID="0f697726-1e6f-4463-b00f-0f4090608380" containerName="revoke-keys"
Oct 30 19:43:01.857918 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:01.857824    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f697726-1e6f-4463-b00f-0f4090608380" containerName="revoke-keys"
Oct 30 19:43:01.857918 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:01.857885    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f697726-1e6f-4463-b00f-0f4090608380" containerName="revoke-keys"
Oct 30 19:43:01.862397 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:01.862377    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"
Oct 30 19:43:01.876366 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:01.876343    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"]
Oct 30 19:43:01.879225 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:01.879199    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:01.880147 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:01.880129    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:43:01.880147 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:01.880139    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:43:01.880278 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:01.880165    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 19:43:01.929911 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:01.929878    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5013423-65b7g\" (UID: \"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"
Oct 30 19:43:01.930119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:01.929931    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkng2\" (UniqueName: \"kubernetes.io/projected/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-kube-api-access-wkng2\") pod \"maas-api-revoke-sub-b5013423-65b7g\" (UID: \"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"
Oct 30 19:43:01.930119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:01.929959    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5013423-65b7g\" (UID: \"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"
Oct 30 19:43:02.030790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:02.030752    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkng2\" (UniqueName: \"kubernetes.io/projected/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-kube-api-access-wkng2\") pod \"maas-api-revoke-sub-b5013423-65b7g\" (UID: \"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"
Oct 30 19:43:02.030987 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:02.030792    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5013423-65b7g\" (UID: \"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"
Oct 30 19:43:02.031089 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:02.031058    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5013423-65b7g\" (UID: \"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"
Oct 30 19:43:02.031618 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:02.031598    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5013423-65b7g\" (UID: \"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"
Oct 30 19:43:02.033483 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:02.033462    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5013423-65b7g\" (UID: \"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"
Oct 30 19:43:02.045227 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:02.045204    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkng2\" (UniqueName: \"kubernetes.io/projected/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-kube-api-access-wkng2\") pod \"maas-api-revoke-sub-b5013423-65b7g\" (UID: \"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"
Oct 30 19:43:02.174954 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:02.174916    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"
Oct 30 19:43:02.473036 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:02.464403    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"]
Oct 30 19:43:02.474384 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:43:02.474349    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1e7edb_32ea_4c0f_8b58_3b7668ae6c57.slice/crio-6f78273f4ed44f7dab87b245eb3e5b297c1ff341b06abc3750716be9dd246f29 WatchSource:0}: Error finding container 6f78273f4ed44f7dab87b245eb3e5b297c1ff341b06abc3750716be9dd246f29: Status 404 returned error can't find the container with id 6f78273f4ed44f7dab87b245eb3e5b297c1ff341b06abc3750716be9dd246f29
Oct 30 19:43:03.462001 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:03.461960    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g" event={"ID":"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57","Type":"ContainerStarted","Data":"bfecb12069982eaa713050e9fb52e5c3d093e764a373e3cf110df4efcd201eee"}
Oct 30 19:43:03.462519 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:03.462074    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g" event={"ID":"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57","Type":"ContainerStarted","Data":"6f78273f4ed44f7dab87b245eb3e5b297c1ff341b06abc3750716be9dd246f29"}
Oct 30 19:43:03.491868 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:03.491789    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g" podStartSLOduration=2.491771605 podStartE2EDuration="2.491771605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:03.488780928 +0000 UTC m=+662.404387784" watchObservedRunningTime="2026-10-30 19:43:03.491771605 +0000 UTC m=+662.407378462"
Oct 30 19:43:04.325030 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:43:04.324994    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:43:04.468506 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:04.468470    6030 generic.go:356] "Generic (PLEG): container finished" podID="8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57" containerID="bfecb12069982eaa713050e9fb52e5c3d093e764a373e3cf110df4efcd201eee" exitCode=0
Oct 30 19:43:04.468870 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:04.468548    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g" event={"ID":"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57","Type":"ContainerDied","Data":"bfecb12069982eaa713050e9fb52e5c3d093e764a373e3cf110df4efcd201eee"}
Oct 30 19:43:05.079825 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.079794    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"]
Oct 30 19:43:05.083570 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.083547    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.095283 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.095254    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8836a8\"/\"e2e-conflict-a-05cbc882-kserve-self-signed-certs\""
Oct 30 19:43:05.096215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.096191    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"]
Oct 30 19:43:05.163951 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.163904    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7de8a961-4f64-4564-bb6c-d517bff87d28-tls-certs\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.164150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.163980    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-tmp-dir\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.164150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.164033    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdvbf\" (UniqueName: \"kubernetes.io/projected/7de8a961-4f64-4564-bb6c-d517bff87d28-kube-api-access-xdvbf\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.164150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.164123    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-model-cache\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.164293 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.164161    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-dshm\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.164293 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.164183    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-home\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.270068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.266620    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7de8a961-4f64-4564-bb6c-d517bff87d28-tls-certs\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.270068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.266693    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-tmp-dir\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.270068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.266745    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdvbf\" (UniqueName: \"kubernetes.io/projected/7de8a961-4f64-4564-bb6c-d517bff87d28-kube-api-access-xdvbf\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.270068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.266787    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-model-cache\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.270068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.266830    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-dshm\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.270068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.266859    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-home\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.270068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.267475    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-home\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.270068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.268477    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-tmp-dir\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.270068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.268722    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-model-cache\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.281268 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.281222    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-dshm\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.293114 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.290112    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdvbf\" (UniqueName: \"kubernetes.io/projected/7de8a961-4f64-4564-bb6c-d517bff87d28-kube-api-access-xdvbf\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.301363 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.301323    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7de8a961-4f64-4564-bb6c-d517bff87d28-tls-certs\") pod \"e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") " pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.399554 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.399516    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:05.610845 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.610818    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"
Oct 30 19:43:05.670471 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.670395    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-maas-api-openshift-ca\") pod \"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57\" (UID: \"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57\") "
Oct 30 19:43:05.670630 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.670513    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-kube-api-access-wkng2\" (UniqueName: \"kubernetes.io/projected/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-kube-api-access-wkng2\") pod \"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57\" (UID: \"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57\") "
Oct 30 19:43:05.670727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.670710    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-maas-api-xks-ca\") pod \"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57\" (UID: \"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57\") "
Oct 30 19:43:05.670826 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.670800    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-maas-api-openshift-ca" pod "8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57" (UID: "8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:05.671020 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.671002    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:05.672710 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.672681    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-kube-api-access-wkng2" pod "8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57" (UID: "8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57"). InnerVolumeSpecName "kube-api-access-wkng2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:05.672788 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.672750    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-maas-api-xks-ca" pod "8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57" (UID: "8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:05.684586 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.684561    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"]
Oct 30 19:43:05.685908 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:43:05.685880    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7de8a961_4f64_4564_bb6c_d517bff87d28.slice/crio-5b5c4a3fd85b49e16e1d14701a18aa3bb5e70bc7f4802d4fe878009bb141ebda WatchSource:0}: Error finding container 5b5c4a3fd85b49e16e1d14701a18aa3bb5e70bc7f4802d4fe878009bb141ebda: Status 404 returned error can't find the container with id 5b5c4a3fd85b49e16e1d14701a18aa3bb5e70bc7f4802d4fe878009bb141ebda
Oct 30 19:43:05.772056 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.772012    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkng2\" (UniqueName: \"kubernetes.io/projected/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-kube-api-access-wkng2\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:05.772179 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:05.772064    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.482740 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:06.482711    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"
Oct 30 19:43:06.482988 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:06.482710    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g" event={"ID":"8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57","Type":"ContainerDied","Data":"6f78273f4ed44f7dab87b245eb3e5b297c1ff341b06abc3750716be9dd246f29"}
Oct 30 19:43:06.482988 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:06.482837    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f78273f4ed44f7dab87b245eb3e5b297c1ff341b06abc3750716be9dd246f29"
Oct 30 19:43:06.484341 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:06.484316    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg" event={"ID":"7de8a961-4f64-4564-bb6c-d517bff87d28","Type":"ContainerStarted","Data":"fcde31d7ee2cb143b85ee44793620237f94a714b01ba1c443bbf8b8377c32db7"}
Oct 30 19:43:06.484469 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:06.484348    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg" event={"ID":"7de8a961-4f64-4564-bb6c-d517bff87d28","Type":"ContainerStarted","Data":"5b5c4a3fd85b49e16e1d14701a18aa3bb5e70bc7f4802d4fe878009bb141ebda"}
Oct 30 19:43:06.504475 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:06.504427    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg" podStartSLOduration=1.504414052 podStartE2EDuration="1.504414052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:06.504305968 +0000 UTC m=+665.419912820" watchObservedRunningTime="2026-10-30 19:43:06.504414052 +0000 UTC m=+665.420020909"
Oct 30 19:43:07.399669 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:07.399634    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:12.118595 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.118555    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"]
Oct 30 19:43:12.124844 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.124816    6030 cpu_manager.go:414] "containerMap: removing container" podUID="8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57" containerName="revoke-keys"
Oct 30 19:43:12.125003 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.124987    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57" containerName="revoke-keys"
Oct 30 19:43:12.125215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.125198    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57" containerName="revoke-keys"
Oct 30 19:43:12.130759 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.130726    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:12.151702 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.151669    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 19:43:12.151987 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.151970    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 19:43:12.152250 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.152234    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 19:43:12.152468 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.152454    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 19:43:12.153726 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.153707    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-84ddec04140b\""
Oct 30 19:43:12.154198 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.154179    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-8p64v\""
Oct 30 19:43:12.219435 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.219391    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71ef5213-5610-4495-9463-9b9cb346e596-provider-credentials\") pod \"payload-processing-external-model-5b47fc8d95-8r2t8\" (UID: \"71ef5213-5610-4495-9463-9b9cb346e596\") " pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:12.219770 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.219746    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71ef5213-5610-4495-9463-9b9cb346e596-routing-overlay-volume\") pod \"payload-processing-external-model-5b47fc8d95-8r2t8\" (UID: \"71ef5213-5610-4495-9463-9b9cb346e596\") " pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:12.220085 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.220065    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6sn6\" (UniqueName: \"kubernetes.io/projected/71ef5213-5610-4495-9463-9b9cb346e596-kube-api-access-s6sn6\") pod \"payload-processing-external-model-5b47fc8d95-8r2t8\" (UID: \"71ef5213-5610-4495-9463-9b9cb346e596\") " pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:12.220315 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.220296    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ef5213-5610-4495-9463-9b9cb346e596-plugins-config-volume\") pod \"payload-processing-external-model-5b47fc8d95-8r2t8\" (UID: \"71ef5213-5610-4495-9463-9b9cb346e596\") " pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:12.237292 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.237261    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"]
Oct 30 19:43:12.328071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.326336    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6sn6\" (UniqueName: \"kubernetes.io/projected/71ef5213-5610-4495-9463-9b9cb346e596-kube-api-access-s6sn6\") pod \"payload-processing-external-model-5b47fc8d95-8r2t8\" (UID: \"71ef5213-5610-4495-9463-9b9cb346e596\") " pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:12.328071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.326408    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ef5213-5610-4495-9463-9b9cb346e596-plugins-config-volume\") pod \"payload-processing-external-model-5b47fc8d95-8r2t8\" (UID: \"71ef5213-5610-4495-9463-9b9cb346e596\") " pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:12.328071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.326469    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71ef5213-5610-4495-9463-9b9cb346e596-provider-credentials\") pod \"payload-processing-external-model-5b47fc8d95-8r2t8\" (UID: \"71ef5213-5610-4495-9463-9b9cb346e596\") " pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:12.328071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.326529    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71ef5213-5610-4495-9463-9b9cb346e596-routing-overlay-volume\") pod \"payload-processing-external-model-5b47fc8d95-8r2t8\" (UID: \"71ef5213-5610-4495-9463-9b9cb346e596\") " pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:12.328071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.327394    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71ef5213-5610-4495-9463-9b9cb346e596-routing-overlay-volume\") pod \"payload-processing-external-model-5b47fc8d95-8r2t8\" (UID: \"71ef5213-5610-4495-9463-9b9cb346e596\") " pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:12.328494 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.328470    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ef5213-5610-4495-9463-9b9cb346e596-plugins-config-volume\") pod \"payload-processing-external-model-5b47fc8d95-8r2t8\" (UID: \"71ef5213-5610-4495-9463-9b9cb346e596\") " pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:12.345613 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.341581    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71ef5213-5610-4495-9463-9b9cb346e596-provider-credentials\") pod \"payload-processing-external-model-5b47fc8d95-8r2t8\" (UID: \"71ef5213-5610-4495-9463-9b9cb346e596\") " pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:12.346788 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.346739    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6sn6\" (UniqueName: \"kubernetes.io/projected/71ef5213-5610-4495-9463-9b9cb346e596-kube-api-access-s6sn6\") pod \"payload-processing-external-model-5b47fc8d95-8r2t8\" (UID: \"71ef5213-5610-4495-9463-9b9cb346e596\") " pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:12.349222 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.349193    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:43:12.349419 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.349391    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg" podUID="715973cd-8d23-4a57-b8ca-d22f81ac95a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:12.497071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.496315    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:12.865493 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:12.865382    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"]
Oct 30 19:43:13.559829 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:13.559791    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8" event={"ID":"71ef5213-5610-4495-9463-9b9cb346e596","Type":"ContainerStarted","Data":"0f82c0aaaa5d27b239fa890de12c230f35004a7d655be35aef048cce441aeb2d"}
Oct 30 19:43:15.400109 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:15.400080    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:15.413250 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:15.413226    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:15.573179 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:15.573139    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8" event={"ID":"71ef5213-5610-4495-9463-9b9cb346e596","Type":"ContainerStarted","Data":"c9279d08194a7c2230e495960a71c30005a131709643d003752c307189468f9b"}
Oct 30 19:43:15.584472 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:15.584444    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:43:15.599379 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:15.599276    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8" podStartSLOduration=1.16956226 podStartE2EDuration="3.599258363s" totalImagesPullingTime="2.429696103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:15.59821007 +0000 UTC m=+674.513816925" watchObservedRunningTime="2026-10-30 19:43:15.599258363 +0000 UTC m=+674.514865219"
Oct 30 19:43:16.579088 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:16.579058    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:16.846005 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:16.845923    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"]
Oct 30 19:43:16.849798 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:16.849759    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:16.869646 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:16.869612    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"]
Oct 30 19:43:16.869884 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:16.869852    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c1fdd6\"/\"distinct-w3-c1fdd6-kserve-self-signed-certs\""
Oct 30 19:43:16.924954 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:16.924914    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrkrr\" (UniqueName: \"kubernetes.io/projected/e7f34b33-c655-42b7-866f-6d61375e0f65-kube-api-access-rrkrr\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:16.925176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:16.924990    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-model-cache\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:16.925176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:16.925157    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-dshm\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:16.925301 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:16.925215    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-home\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:16.925301 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:16.925278    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-tmp-dir\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:16.925437 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:16.925411    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f34b33-c655-42b7-866f-6d61375e0f65-tls-certs\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:17.036134 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:17.032622    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrkrr\" (UniqueName: \"kubernetes.io/projected/e7f34b33-c655-42b7-866f-6d61375e0f65-kube-api-access-rrkrr\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:17.036134 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:17.032715    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-model-cache\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:17.036134 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:17.032858    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-dshm\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:17.036134 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:17.032910    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-home\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:17.036134 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:17.032937    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-tmp-dir\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:17.036134 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:17.032980    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f34b33-c655-42b7-866f-6d61375e0f65-tls-certs\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:17.036134 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:17.034442    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-home\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:17.036134 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:17.035020    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-tmp-dir\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:17.036134 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:17.035298    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-model-cache\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:17.037300 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:17.037272    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-dshm\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:17.047102 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:17.044642    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f34b33-c655-42b7-866f-6d61375e0f65-tls-certs\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:17.054865 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:17.054837    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrkrr\" (UniqueName: \"kubernetes.io/projected/e7f34b33-c655-42b7-866f-6d61375e0f65-kube-api-access-rrkrr\") pod \"distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") " pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:17.175226 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:17.175192    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:17.453189 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:17.453160    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"]
Oct 30 19:43:17.454411 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:43:17.454385    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7f34b33_c655_42b7_866f_6d61375e0f65.slice/crio-1e6c2da006852c471d8331d576cd8975b7e583e9e2345fbce2ba695f5edf7476 WatchSource:0}: Error finding container 1e6c2da006852c471d8331d576cd8975b7e583e9e2345fbce2ba695f5edf7476: Status 404 returned error can't find the container with id 1e6c2da006852c471d8331d576cd8975b7e583e9e2345fbce2ba695f5edf7476
Oct 30 19:43:17.617565 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:17.617471    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" event={"ID":"e7f34b33-c655-42b7-866f-6d61375e0f65","Type":"ContainerStarted","Data":"f430a587d812c0287f1396b41c53274c8994242b4ca6c67fdf7c80fd9125a46e"}
Oct 30 19:43:17.617565 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:17.617538    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" event={"ID":"e7f34b33-c655-42b7-866f-6d61375e0f65","Type":"ContainerStarted","Data":"1e6c2da006852c471d8331d576cd8975b7e583e9e2345fbce2ba695f5edf7476"}
Oct 30 19:43:17.645319 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:17.645270    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podStartSLOduration=1.64525579 podStartE2EDuration="1.64525579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:17.644279914 +0000 UTC m=+676.559886771" watchObservedRunningTime="2026-10-30 19:43:17.64525579 +0000 UTC m=+676.560862646"
Oct 30 19:43:18.175359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:18.175324    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:18.972285 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:18.972256    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"]
Oct 30 19:43:18.976065 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:18.976031    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:18.986929 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:18.986906    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f06af6\"/\"distinct-w2-f06af6-kserve-self-signed-certs\""
Oct 30 19:43:18.990073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:18.990026    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"]
Oct 30 19:43:19.051069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.051010    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-tls-certs\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.051273 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.051106    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-tmp-dir\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.051273 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.051145    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-home\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.051273 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.051196    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57wc7\" (UniqueName: \"kubernetes.io/projected/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-kube-api-access-57wc7\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.051447 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.051294    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-model-cache\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.051447 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.051361    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-dshm\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.166069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.165445    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57wc7\" (UniqueName: \"kubernetes.io/projected/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-kube-api-access-57wc7\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.166069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.165894    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-model-cache\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.166069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.165949    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-dshm\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.166069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.166009    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-tls-certs\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.166069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.166075    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-tmp-dir\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.166475 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.166136    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-home\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.170070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.166615    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-home\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.170070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.166892    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-model-cache\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.170070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.167498    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-tmp-dir\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.174068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.171139    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-dshm\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.174068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.173717    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-tls-certs\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.195322 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.185948    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57wc7\" (UniqueName: \"kubernetes.io/projected/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-kube-api-access-57wc7\") pod \"distinct-w2-f06af6-kserve-6d647f857b-xgpmw\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.290881 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.290799    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:19.587483 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.587381    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"]
Oct 30 19:43:19.625472 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:19.625444    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" event={"ID":"0772e44f-03d9-4cf5-a642-c0e0a50b27e9","Type":"ContainerStarted","Data":"2c15749b45e223e3b44f905bd8de201551f15ba493dfbbdbe93b0a26699d512b"}
Oct 30 19:43:20.634782 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:20.634729    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" event={"ID":"0772e44f-03d9-4cf5-a642-c0e0a50b27e9","Type":"ContainerStarted","Data":"8aba076b8f671e9a06c675594b74e6408b1883c4c7d6e9cc157bbb4592659544"}
Oct 30 19:43:20.656825 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:20.656769    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podStartSLOduration=2.65675641 podStartE2EDuration="2.65675641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:20.65558844 +0000 UTC m=+679.571195289" watchObservedRunningTime="2026-10-30 19:43:20.65675641 +0000 UTC m=+679.572363278"
Oct 30 19:43:21.291438 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:21.291402    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:21.892404 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:21.892367    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"]
Oct 30 19:43:21.896093 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:21.896070    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:21.908258 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:21.908235    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f06af6-openshift-default-dockercfg-tcjql\""
Oct 30 19:43:21.910643 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:21.910617    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"]
Oct 30 19:43:21.913627 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:21.913599    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:21.913764 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:21.913639    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:21.913764 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:21.913715    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:21.913764 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:21.913748    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrw29\" (UniqueName: \"kubernetes.io/projected/409380a1-684a-4be3-9cfc-a9e72c247409-kube-api-access-mrw29\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:21.913933 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:21.913788    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409380a1-684a-4be3-9cfc-a9e72c247409-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:21.913933 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:21.913815    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:21.913933 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:21.913898    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409380a1-684a-4be3-9cfc-a9e72c247409-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:21.913933 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:21.913921    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409380a1-684a-4be3-9cfc-a9e72c247409-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:21.914239 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:21.913946    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.014490 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.014450    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409380a1-684a-4be3-9cfc-a9e72c247409-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.014672 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.014496    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409380a1-684a-4be3-9cfc-a9e72c247409-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.014672 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.014535    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.014672 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.014615    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.014672 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.014644    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.014898 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.014697    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.014898 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.014742    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrw29\" (UniqueName: \"kubernetes.io/projected/409380a1-684a-4be3-9cfc-a9e72c247409-kube-api-access-mrw29\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.014898 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.014821    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409380a1-684a-4be3-9cfc-a9e72c247409-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.014898 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.014861    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.015124 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.015000    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.015124 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.015033    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.015274 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.015186    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.015343 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.015284    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.015536 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.015520    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409380a1-684a-4be3-9cfc-a9e72c247409-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.017125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.017103    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.017294 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.017277    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409380a1-684a-4be3-9cfc-a9e72c247409-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.028450 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.028428    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409380a1-684a-4be3-9cfc-a9e72c247409-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.028640 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.028617    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrw29\" (UniqueName: \"kubernetes.io/projected/409380a1-684a-4be3-9cfc-a9e72c247409-kube-api-access-mrw29\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.211746 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.211666    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:22.506058 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.506020    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"]
Oct 30 19:43:22.506893 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:43:22.506869    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409380a1_684a_4be3_9cfc_a9e72c247409.slice/crio-4f0068b30c28587fff01c680e2227adc202be305cab6242309bcbc78d4bde2a1 WatchSource:0}: Error finding container 4f0068b30c28587fff01c680e2227adc202be305cab6242309bcbc78d4bde2a1: Status 404 returned error can't find the container with id 4f0068b30c28587fff01c680e2227adc202be305cab6242309bcbc78d4bde2a1
Oct 30 19:43:22.509367 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.509336    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:43:22.509442 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.509400    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:43:22.509442 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.509428    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:43:22.664075 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.663348    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9" event={"ID":"409380a1-684a-4be3-9cfc-a9e72c247409","Type":"ContainerStarted","Data":"49f2cb1762d21c727d9619f77571c7832d10842769c9806268122b190c9ac855"}
Oct 30 19:43:22.664075 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:22.663417    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9" event={"ID":"409380a1-684a-4be3-9cfc-a9e72c247409","Type":"ContainerStarted","Data":"4f0068b30c28587fff01c680e2227adc202be305cab6242309bcbc78d4bde2a1"}
Oct 30 19:43:23.758484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:23.758406    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9" podStartSLOduration=2.758387085 podStartE2EDuration="2.758387085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:23.747974127 +0000 UTC m=+682.663580985" watchObservedRunningTime="2026-10-30 19:43:23.758387085 +0000 UTC m=+682.673993942"
Oct 30 19:43:24.214255 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:24.213989    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:24.214974 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:24.214931    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:43:24.215328 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:24.215025    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9" podUID="409380a1-684a-4be3-9cfc-a9e72c247409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 19:43:25.225176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:25.225139    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:43:25.225777 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:25.225745    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9" podUID="409380a1-684a-4be3-9cfc-a9e72c247409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 19:43:25.580126 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:25.579139    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg" podUID="7de8a961-4f64-4564-bb6c-d517bff87d28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:25.622171 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:25.622127    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:25.862139 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:25.859151    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:25.893070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:25.890146    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 19:43:26.011071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:26.005172    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:26.217071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:26.214289    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:43:26.217071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:26.214348    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9" podUID="409380a1-684a-4be3-9cfc-a9e72c247409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 19:43:27.178070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:27.177450    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:27.212468 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:27.212434    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:43:27.212731 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:27.212702    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9" podUID="409380a1-684a-4be3-9cfc-a9e72c247409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 19:43:27.235782 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:27.235754    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:27.617394 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:27.617316    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:27.741476 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:27.741448    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:43:28.212107 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:28.212071    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:43:28.212507 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:28.212121    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9" podUID="409380a1-684a-4be3-9cfc-a9e72c247409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 19:43:29.212232 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:29.212189    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:43:29.212672 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:29.212257    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9" podUID="409380a1-684a-4be3-9cfc-a9e72c247409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 19:43:29.291024 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:29.290985    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:29.310165 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:29.310139    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:29.753188 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:29.753159    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:43:30.228907 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:30.228868    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:30.230998 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:30.230973    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:30.232563 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:30.232542    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 19:43:34.287823 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:43:34.287786    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:43:46.282839 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.282802    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"]
Oct 30 19:43:46.289177 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.289155    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.300210 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.300182    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"]
Oct 30 19:43:46.325598 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.325558    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6c8x\" (UniqueName: \"kubernetes.io/projected/2c441eb5-f8ed-413a-b77c-4a8e043c431e-kube-api-access-k6c8x\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.325778 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.325622    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.325778 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.325651    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.325778 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.325757    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.325958 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.325836    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.325958 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.325869    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.325958 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.325890    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.325958 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.325909    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.326203 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.325970    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.427289 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.427236    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.427488 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.427362    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6c8x\" (UniqueName: \"kubernetes.io/projected/2c441eb5-f8ed-413a-b77c-4a8e043c431e-kube-api-access-k6c8x\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.427488 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.427401    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.427488 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.427428    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.427488 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.427474    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.427700 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.427538    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.427700 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.427571    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.427700 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.427605    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.427700 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.427631    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.428175 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.428144    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.428353 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.428172    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.428353 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.428343    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.428499 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.428264    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.428781 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.428741    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.430190 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.430163    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.430490 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.430472    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.441156 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.441134    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6c8x\" (UniqueName: \"kubernetes.io/projected/2c441eb5-f8ed-413a-b77c-4a8e043c431e-kube-api-access-k6c8x\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.441474 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.441456    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.604246 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.604153    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:46.894334 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.894300    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"]
Oct 30 19:43:46.897901 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:43:46.897865    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c441eb5_f8ed_413a_b77c_4a8e043c431e.slice/crio-55938ff5d133fc722e43d4fb6cf99261222b46983bc00600c883fa6f1b6bb28d WatchSource:0}: Error finding container 55938ff5d133fc722e43d4fb6cf99261222b46983bc00600c883fa6f1b6bb28d: Status 404 returned error can't find the container with id 55938ff5d133fc722e43d4fb6cf99261222b46983bc00600c883fa6f1b6bb28d
Oct 30 19:43:46.900730 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.900694    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:43:46.900857 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.900782    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:43:46.900857 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:46.900830    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:43:47.859746 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:47.859694    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz" event={"ID":"2c441eb5-f8ed-413a-b77c-4a8e043c431e","Type":"ContainerStarted","Data":"c3f716db95a5b245a94ac8f5edf371a43b278233b1931af4ebdbe6b64bdafac3"}
Oct 30 19:43:47.860261 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:47.859764    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz" event={"ID":"2c441eb5-f8ed-413a-b77c-4a8e043c431e","Type":"ContainerStarted","Data":"55938ff5d133fc722e43d4fb6cf99261222b46983bc00600c883fa6f1b6bb28d"}
Oct 30 19:43:47.890697 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:47.890630    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz" podStartSLOduration=1.8906123 podStartE2EDuration="1.8906123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:47.888537289 +0000 UTC m=+706.804144146" watchObservedRunningTime="2026-10-30 19:43:47.8906123 +0000 UTC m=+706.806219157"
Oct 30 19:43:47.968350 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:47.968313    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"]
Oct 30 19:43:47.999068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:47.998300    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.006128 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.000203    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"]
Oct 30 19:43:48.010297 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.009334    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f06af6\"/\"distinct-2-w2-f06af6-kserve-self-signed-certs\""
Oct 30 19:43:48.045843 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.045815    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37429652-9f2c-4abc-b27a-598fad5ab338-tls-certs\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.046006 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.045865    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-home\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.046006 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.045895    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n24k9\" (UniqueName: \"kubernetes.io/projected/37429652-9f2c-4abc-b27a-598fad5ab338-kube-api-access-n24k9\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.046149 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.046007    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-tmp-dir\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.046210 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.046156    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-dshm\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.046210 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.046185    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-model-cache\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.159074 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.155509    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37429652-9f2c-4abc-b27a-598fad5ab338-tls-certs\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.159074 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.156188    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-home\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.159074 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.156222    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n24k9\" (UniqueName: \"kubernetes.io/projected/37429652-9f2c-4abc-b27a-598fad5ab338-kube-api-access-n24k9\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.159074 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.156258    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-tmp-dir\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.159074 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.156397    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-dshm\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.159074 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.156419    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-model-cache\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.159074 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.156837    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-model-cache\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.159074 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.157106    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-home\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.159074 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.157679    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-tmp-dir\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.196308 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.186910    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-dshm\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.219070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.211761    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n24k9\" (UniqueName: \"kubernetes.io/projected/37429652-9f2c-4abc-b27a-598fad5ab338-kube-api-access-n24k9\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.219070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.211880    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37429652-9f2c-4abc-b27a-598fad5ab338-tls-certs\") pod \"distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") " pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.334083 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.334007    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:48.612922 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.612116    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:48.612922 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.612188    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:43:48.612922 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.612247    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz" podUID="2c441eb5-f8ed-413a-b77c-4a8e043c431e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 19:43:48.675624 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.674829    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"]
Oct 30 19:43:48.692594 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:43:48.692566    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37429652_9f2c_4abc_b27a_598fad5ab338.slice/crio-e2ac84ef03d853b0216faf35d31ebdee6d51974787e09dc6d68ee6c68a7589bd WatchSource:0}: Error finding container e2ac84ef03d853b0216faf35d31ebdee6d51974787e09dc6d68ee6c68a7589bd: Status 404 returned error can't find the container with id e2ac84ef03d853b0216faf35d31ebdee6d51974787e09dc6d68ee6c68a7589bd
Oct 30 19:43:48.872755 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:48.872718    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" event={"ID":"37429652-9f2c-4abc-b27a-598fad5ab338","Type":"ContainerStarted","Data":"e2ac84ef03d853b0216faf35d31ebdee6d51974787e09dc6d68ee6c68a7589bd"}
Oct 30 19:43:49.617074 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:49.612373    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:43:49.617074 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:49.612435    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz" podUID="2c441eb5-f8ed-413a-b77c-4a8e043c431e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 19:43:49.908884 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:49.908790    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" event={"ID":"37429652-9f2c-4abc-b27a-598fad5ab338","Type":"ContainerStarted","Data":"8cfb8bc3d11d8b4c7a8b51f3b4b68a13b44f47872dd11af4274a347e16b5a20f"}
Oct 30 19:43:49.945973 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:49.940814    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podStartSLOduration=2.9407933 podStartE2EDuration="2.9407933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:49.937414084 +0000 UTC m=+708.853020943" watchObservedRunningTime="2026-10-30 19:43:49.9407933 +0000 UTC m=+708.856400157"
Oct 30 19:43:50.340125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:50.339329    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:50.635126 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:50.613032    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"]
Oct 30 19:43:50.635126 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:50.614411    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:43:50.635126 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:50.614454    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz" podUID="2c441eb5-f8ed-413a-b77c-4a8e043c431e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 19:43:51.607070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:51.606432    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:43:51.607070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:51.606505    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz" podUID="2c441eb5-f8ed-413a-b77c-4a8e043c431e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 19:43:51.619068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:51.616110    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:51.756002 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:51.755461    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:43:51.937037 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:51.918123    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 19:43:51.941176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:51.940337    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:52.612244 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:52.611423    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:43:52.612244 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:52.611485    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz" podUID="2c441eb5-f8ed-413a-b77c-4a8e043c431e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 19:43:53.606068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:53.604728    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:43:53.606068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:53.604801    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz" podUID="2c441eb5-f8ed-413a-b77c-4a8e043c431e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 19:43:55.613843 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:55.613774    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8" podUID="71ef5213-5610-4495-9463-9b9cb346e596" containerName="payload-processing" containerID="cri-o://c9279d08194a7c2230e495960a71c30005a131709643d003752c307189468f9b" gracePeriod=25
Oct 30 19:43:55.648211 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:55.648178    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:43:55.648303 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:55.648247    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz" podUID="2c441eb5-f8ed-413a-b77c-4a8e043c431e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:55.817883 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:55.817853    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:55.819774 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:55.819687    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:55.820873 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:55.820824    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 19:43:55.915938 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:55.915909    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:56.008279 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.008244    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ef5213-5610-4495-9463-9b9cb346e596-provider-credentials\" (UniqueName: \"kubernetes.io/projected/71ef5213-5610-4495-9463-9b9cb346e596-provider-credentials\") pod \"71ef5213-5610-4495-9463-9b9cb346e596\" (UID: \"71ef5213-5610-4495-9463-9b9cb346e596\") "
Oct 30 19:43:56.008455 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.008331    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ef5213-5610-4495-9463-9b9cb346e596-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71ef5213-5610-4495-9463-9b9cb346e596-routing-overlay-volume\") pod \"71ef5213-5610-4495-9463-9b9cb346e596\" (UID: \"71ef5213-5610-4495-9463-9b9cb346e596\") "
Oct 30 19:43:56.008455 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.008369    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ef5213-5610-4495-9463-9b9cb346e596-kube-api-access-s6sn6\" (UniqueName: \"kubernetes.io/projected/71ef5213-5610-4495-9463-9b9cb346e596-kube-api-access-s6sn6\") pod \"71ef5213-5610-4495-9463-9b9cb346e596\" (UID: \"71ef5213-5610-4495-9463-9b9cb346e596\") "
Oct 30 19:43:56.008543 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.008500    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ef5213-5610-4495-9463-9b9cb346e596-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ef5213-5610-4495-9463-9b9cb346e596-plugins-config-volume\") pod \"71ef5213-5610-4495-9463-9b9cb346e596\" (UID: \"71ef5213-5610-4495-9463-9b9cb346e596\") "
Oct 30 19:43:56.008807 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.008782    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ef5213-5610-4495-9463-9b9cb346e596-routing-overlay-volume" pod "71ef5213-5610-4495-9463-9b9cb346e596" (UID: "71ef5213-5610-4495-9463-9b9cb346e596"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:56.009073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.008877    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ef5213-5610-4495-9463-9b9cb346e596-plugins-config-volume" pod "71ef5213-5610-4495-9463-9b9cb346e596" (UID: "71ef5213-5610-4495-9463-9b9cb346e596"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:56.010758 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.010723    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ef5213-5610-4495-9463-9b9cb346e596-kube-api-access-s6sn6" pod "71ef5213-5610-4495-9463-9b9cb346e596" (UID: "71ef5213-5610-4495-9463-9b9cb346e596"). InnerVolumeSpecName "kube-api-access-s6sn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:56.010852 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.010754    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ef5213-5610-4495-9463-9b9cb346e596-provider-credentials" pod "71ef5213-5610-4495-9463-9b9cb346e596" (UID: "71ef5213-5610-4495-9463-9b9cb346e596"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:56.058433 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.058401    6030 generic.go:356] "Generic (PLEG): container finished" podID="71ef5213-5610-4495-9463-9b9cb346e596" containerID="c9279d08194a7c2230e495960a71c30005a131709643d003752c307189468f9b" exitCode=0
Oct 30 19:43:56.058620 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.058445    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"
Oct 30 19:43:56.058620 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.058500    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8" event={"ID":"71ef5213-5610-4495-9463-9b9cb346e596","Type":"ContainerDied","Data":"c9279d08194a7c2230e495960a71c30005a131709643d003752c307189468f9b"}
Oct 30 19:43:56.058620 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.058560    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8" event={"ID":"71ef5213-5610-4495-9463-9b9cb346e596","Type":"ContainerDied","Data":"0f82c0aaaa5d27b239fa890de12c230f35004a7d655be35aef048cce441aeb2d"}
Oct 30 19:43:56.058620 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.058589    6030 scope.go:122] "RemoveContainer" containerID="c9279d08194a7c2230e495960a71c30005a131709643d003752c307189468f9b"
Oct 30 19:43:56.072007 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.071990    6030 scope.go:122] "RemoveContainer" containerID="c9279d08194a7c2230e495960a71c30005a131709643d003752c307189468f9b"
Oct 30 19:43:56.072337 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:43:56.072312    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9279d08194a7c2230e495960a71c30005a131709643d003752c307189468f9b\": container with ID starting with c9279d08194a7c2230e495960a71c30005a131709643d003752c307189468f9b not found: ID does not exist" containerID="c9279d08194a7c2230e495960a71c30005a131709643d003752c307189468f9b"
Oct 30 19:43:56.072386 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.072354    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9279d08194a7c2230e495960a71c30005a131709643d003752c307189468f9b"} err="failed to get container status \"c9279d08194a7c2230e495960a71c30005a131709643d003752c307189468f9b\": rpc error: code = NotFound desc = could not find container \"c9279d08194a7c2230e495960a71c30005a131709643d003752c307189468f9b\": container with ID starting with c9279d08194a7c2230e495960a71c30005a131709643d003752c307189468f9b not found: ID does not exist"
Oct 30 19:43:56.084412 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.084390    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"]
Oct 30 19:43:56.088938 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.088911    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b47fc8d95-8r2t8"]
Oct 30 19:43:56.110185 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.110156    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71ef5213-5610-4495-9463-9b9cb346e596-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.110185 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.110184    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6sn6\" (UniqueName: \"kubernetes.io/projected/71ef5213-5610-4495-9463-9b9cb346e596-kube-api-access-s6sn6\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.110347 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.110200    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ef5213-5610-4495-9463-9b9cb346e596-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.110347 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.110215    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71ef5213-5610-4495-9463-9b9cb346e596-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.583059 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.583012    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"]
Oct 30 19:43:56.583540 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.583520    6030 cpu_manager.go:414] "containerMap: removing container" podUID="71ef5213-5610-4495-9463-9b9cb346e596" containerName="payload-processing"
Oct 30 19:43:56.583635 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.583542    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ef5213-5610-4495-9463-9b9cb346e596" containerName="payload-processing"
Oct 30 19:43:56.583720 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.583706    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ef5213-5610-4495-9463-9b9cb346e596" containerName="payload-processing"
Oct 30 19:43:56.588489 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.588464    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.615163 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.615125    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.619228 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.615750    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.619228 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.615787    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjtsw\" (UniqueName: \"kubernetes.io/projected/becea00a-6860-4f4e-830f-6f504d0cf8b8-kube-api-access-pjtsw\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.619228 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.615863    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/becea00a-6860-4f4e-830f-6f504d0cf8b8-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.619228 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.615917    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.619228 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.615958    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.619228 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.616018    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.619228 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.616066    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.619228 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.616093    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.619228 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.616300    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"]
Oct 30 19:43:56.717220 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.717184    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.717220 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.717221    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.717492 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.717243    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.717492 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.717292    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.717492 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.717313    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.717492 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.717328    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjtsw\" (UniqueName: \"kubernetes.io/projected/becea00a-6860-4f4e-830f-6f504d0cf8b8-kube-api-access-pjtsw\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.717492 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.717394    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/becea00a-6860-4f4e-830f-6f504d0cf8b8-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.717492 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.717444    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.717492 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.717481    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.718031 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.717972    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.718325 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.718300    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.718420 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.718341    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/becea00a-6860-4f4e-830f-6f504d0cf8b8-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.718420 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.718369    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.718420 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.718343    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.720437 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.720411    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.720529 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.720514    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.734978 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.734949    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjtsw\" (UniqueName: \"kubernetes.io/projected/becea00a-6860-4f4e-830f-6f504d0cf8b8-kube-api-access-pjtsw\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.736686 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.736663    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:56.904835 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:56.904800    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:57.173122 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:57.173014    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"]
Oct 30 19:43:57.204749 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:57.204715    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ef5213-5610-4495-9463-9b9cb346e596" path="/var/lib/kubelet/pods/71ef5213-5610-4495-9463-9b9cb346e596/volumes"
Oct 30 19:43:57.205163 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:57.205140    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"]
Oct 30 19:43:57.211556 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:57.211513    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:43:57.211747 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:57.211718    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:43:57.211841 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:57.211784    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:43:58.092529 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:58.092489    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh" event={"ID":"becea00a-6860-4f4e-830f-6f504d0cf8b8","Type":"ContainerStarted","Data":"5fa8209ac1607463dc5f87b61c38b35342d0fb77ae0124f27cbb80f835799ca1"}
Oct 30 19:43:58.093135 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:58.093109    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh" event={"ID":"becea00a-6860-4f4e-830f-6f504d0cf8b8","Type":"ContainerStarted","Data":"7a8b4d5a24acb62c1a27f3d835098dae1411b86a707b9b16bc4de89ae425a6da"}
Oct 30 19:43:58.123662 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:58.123609    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh" podStartSLOduration=2.123594363 podStartE2EDuration="2.123594363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:58.122789321 +0000 UTC m=+717.038396159" watchObservedRunningTime="2026-10-30 19:43:58.123594363 +0000 UTC m=+717.039201270"
Oct 30 19:43:58.335192 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:58.335158    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:58.347970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:58.347902    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:58.905025 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:58.904996    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:43:58.906413 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:58.906390    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:43:58.906540 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:58.906440    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh" podUID="becea00a-6860-4f4e-830f-6f504d0cf8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 19:43:59.111980 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:59.111954    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:43:59.905382 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:59.905346    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:43:59.905565 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:43:59.905407    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh" podUID="becea00a-6860-4f4e-830f-6f504d0cf8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 19:44:00.905132 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:00.905100    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:44:00.905615 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:00.905149    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh" podUID="becea00a-6860-4f4e-830f-6f504d0cf8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 19:44:01.909503 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:01.909469    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:44:01.910127 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:01.910096    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh" podUID="becea00a-6860-4f4e-830f-6f504d0cf8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 19:44:02.907110 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:02.906559    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:44:02.907110 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:02.906625    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh" podUID="becea00a-6860-4f4e-830f-6f504d0cf8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 19:44:03.905201 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:03.905169    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:44:03.905594 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:03.905230    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh" podUID="becea00a-6860-4f4e-830f-6f504d0cf8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 19:44:04.288831 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:44:04.288736    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:44:05.867076 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:05.865624    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:44:05.867076 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:05.866874    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:44:05.869178 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:05.868116    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 19:44:07.857110 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:07.857070    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"]
Oct 30 19:44:07.863577 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:07.863529    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:07.871196 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:07.871134    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"]
Oct 30 19:44:07.871592 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:07.871567    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8836a8\"/\"distinct-w0-8836a8-kserve-self-signed-certs\""
Oct 30 19:44:07.936567 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:07.936527    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-home\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:07.936748 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:07.936723    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3be41ac-1e27-4393-a083-c8a64286d53f-tls-certs\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:07.936806 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:07.936774    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-tmp-dir\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:07.936909 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:07.936890    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-dshm\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:07.936962 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:07.936921    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-model-cache\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:07.936962 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:07.936943    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b5gn\" (UniqueName: \"kubernetes.io/projected/a3be41ac-1e27-4393-a083-c8a64286d53f-kube-api-access-7b5gn\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:08.039895 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:08.039846    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3be41ac-1e27-4393-a083-c8a64286d53f-tls-certs\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:08.040602 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:08.040570    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-tmp-dir\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:08.040872 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:08.040851    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-dshm\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:08.040989 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:08.040975    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-model-cache\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:08.041117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:08.041101    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b5gn\" (UniqueName: \"kubernetes.io/projected/a3be41ac-1e27-4393-a083-c8a64286d53f-kube-api-access-7b5gn\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:08.041272 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:08.041259    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-home\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:08.041878 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:08.041856    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-home\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:08.060069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:08.042493    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-tmp-dir\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:08.060281 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:08.042658    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-model-cache\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:08.060426 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:08.054136    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3be41ac-1e27-4393-a083-c8a64286d53f-tls-certs\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:08.081407 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:08.081365    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-dshm\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:08.083005 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:08.082979    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b5gn\" (UniqueName: \"kubernetes.io/projected/a3be41ac-1e27-4393-a083-c8a64286d53f-kube-api-access-7b5gn\") pod \"distinct-w0-8836a8-kserve-7c66db8877-zldpk\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:08.201096 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:08.197322    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:08.488093 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:08.488065    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"]
Oct 30 19:44:08.489444 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:44:08.489416    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3be41ac_1e27_4393_a083_c8a64286d53f.slice/crio-a27efb62dd72d22d4f2fab80b5dbace0c9a63e4835646a1b916eb21d5570e643 WatchSource:0}: Error finding container a27efb62dd72d22d4f2fab80b5dbace0c9a63e4835646a1b916eb21d5570e643: Status 404 returned error can't find the container with id a27efb62dd72d22d4f2fab80b5dbace0c9a63e4835646a1b916eb21d5570e643
Oct 30 19:44:09.288067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:09.285543    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" event={"ID":"a3be41ac-1e27-4393-a083-c8a64286d53f","Type":"ContainerStarted","Data":"76ef8f53ed0b46e2612ad4d80f5aac87888b61e099f13bf6c18baa3568973bd7"}
Oct 30 19:44:09.288067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:09.285603    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" event={"ID":"a3be41ac-1e27-4393-a083-c8a64286d53f","Type":"ContainerStarted","Data":"a27efb62dd72d22d4f2fab80b5dbace0c9a63e4835646a1b916eb21d5570e643"}
Oct 30 19:44:09.312758 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:09.312706    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podStartSLOduration=2.312691773 podStartE2EDuration="2.312691773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:09.311027981 +0000 UTC m=+728.226634838" watchObservedRunningTime="2026-10-30 19:44:09.312691773 +0000 UTC m=+728.228298631"
Oct 30 19:44:10.201472 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:10.201438    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:11.651662 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.651622    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"]
Oct 30 19:44:11.655439 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.655421    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.702345 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.702313    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.702462 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.702373    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.702504 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.702464    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.702542 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.702528    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.702576 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.702548    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c76eb36-5c29-41fa-8289-d8f08fba2181-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.702609 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.702581    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.702674 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.702652    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.702766 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.702753    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.702849 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.702835    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9bzn\" (UniqueName: \"kubernetes.io/projected/8c76eb36-5c29-41fa-8289-d8f08fba2181-kube-api-access-r9bzn\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.714788 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.714759    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"]
Oct 30 19:44:11.804518 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.804484    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.804729 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.804566    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.804729 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.804606    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.804729 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.804663    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9bzn\" (UniqueName: \"kubernetes.io/projected/8c76eb36-5c29-41fa-8289-d8f08fba2181-kube-api-access-r9bzn\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.804729 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.804680    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.804957 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.804733    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.804957 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.804777    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.804957 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.804830    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.804957 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.804855    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c76eb36-5c29-41fa-8289-d8f08fba2181-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.805187 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.805153    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.805240 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.805213    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.805293 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.805255    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.805527 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.805471    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.805610 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.805565    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c76eb36-5c29-41fa-8289-d8f08fba2181-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.807294 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.807271    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.807394 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.807346    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.818557 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.818532    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9bzn\" (UniqueName: \"kubernetes.io/projected/8c76eb36-5c29-41fa-8289-d8f08fba2181-kube-api-access-r9bzn\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.818990 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.818969    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:11.969227 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:11.969144    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:12.197168 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.197119    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg" podUID="7de8a961-4f64-4564-bb6c-d517bff87d28" containerName="main" containerID="cri-o://fcde31d7ee2cb143b85ee44793620237f94a714b01ba1c443bbf8b8377c32db7" gracePeriod=45
Oct 30 19:44:12.256941 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.256901    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"]
Oct 30 19:44:12.259014 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:44:12.258986    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c76eb36_5c29_41fa_8289_d8f08fba2181.slice/crio-5483220e66417f21cfc33d47d3996b9c112af615cdf1da2ce3b5ea82d3cef922 WatchSource:0}: Error finding container 5483220e66417f21cfc33d47d3996b9c112af615cdf1da2ce3b5ea82d3cef922: Status 404 returned error can't find the container with id 5483220e66417f21cfc33d47d3996b9c112af615cdf1da2ce3b5ea82d3cef922
Oct 30 19:44:12.302627 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.302587    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:44:12.302754 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.302681    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:44:12.302805 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.302767    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:44:12.528086 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.528000    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:44:12.616284 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.616251    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-model-cache\") pod \"7de8a961-4f64-4564-bb6c-d517bff87d28\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") "
Oct 30 19:44:12.616468 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.616391    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-dshm\") pod \"7de8a961-4f64-4564-bb6c-d517bff87d28\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") "
Oct 30 19:44:12.616468 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.616430    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7de8a961-4f64-4564-bb6c-d517bff87d28-tls-certs\" (UniqueName: \"kubernetes.io/secret/7de8a961-4f64-4564-bb6c-d517bff87d28-tls-certs\") pod \"7de8a961-4f64-4564-bb6c-d517bff87d28\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") "
Oct 30 19:44:12.616468 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.616461    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-tmp-dir\") pod \"7de8a961-4f64-4564-bb6c-d517bff87d28\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") "
Oct 30 19:44:12.616624 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.616550    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-model-cache" pod "7de8a961-4f64-4564-bb6c-d517bff87d28" (UID: "7de8a961-4f64-4564-bb6c-d517bff87d28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:12.616624 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.616572    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de8a961-4f64-4564-bb6c-d517bff87d28-kube-api-access-xdvbf\" (UniqueName: \"kubernetes.io/projected/7de8a961-4f64-4564-bb6c-d517bff87d28-kube-api-access-xdvbf\") pod \"7de8a961-4f64-4564-bb6c-d517bff87d28\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") "
Oct 30 19:44:12.616624 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.616609    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-home\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-home\") pod \"7de8a961-4f64-4564-bb6c-d517bff87d28\" (UID: \"7de8a961-4f64-4564-bb6c-d517bff87d28\") "
Oct 30 19:44:12.616774 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.616751    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-tmp-dir" pod "7de8a961-4f64-4564-bb6c-d517bff87d28" (UID: "7de8a961-4f64-4564-bb6c-d517bff87d28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:12.617024 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.617009    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:12.617024 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.617015    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-home" pod "7de8a961-4f64-4564-bb6c-d517bff87d28" (UID: "7de8a961-4f64-4564-bb6c-d517bff87d28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:12.617161 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.617028    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:12.619305 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.619279    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de8a961-4f64-4564-bb6c-d517bff87d28-kube-api-access-xdvbf" pod "7de8a961-4f64-4564-bb6c-d517bff87d28" (UID: "7de8a961-4f64-4564-bb6c-d517bff87d28"). InnerVolumeSpecName "kube-api-access-xdvbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:12.619429 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.619320    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-dshm" pod "7de8a961-4f64-4564-bb6c-d517bff87d28" (UID: "7de8a961-4f64-4564-bb6c-d517bff87d28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:12.619558 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.619538    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7de8a961-4f64-4564-bb6c-d517bff87d28-tls-certs" pod "7de8a961-4f64-4564-bb6c-d517bff87d28" (UID: "7de8a961-4f64-4564-bb6c-d517bff87d28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:12.718636 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.718607    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdvbf\" (UniqueName: \"kubernetes.io/projected/7de8a961-4f64-4564-bb6c-d517bff87d28-kube-api-access-xdvbf\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:12.718636 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.718633    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:12.718636 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.718643    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7de8a961-4f64-4564-bb6c-d517bff87d28-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:12.719137 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:12.718652    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7de8a961-4f64-4564-bb6c-d517bff87d28-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:13.311636 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:13.311596    6030 generic.go:356] "Generic (PLEG): container finished" podID="7de8a961-4f64-4564-bb6c-d517bff87d28" containerID="fcde31d7ee2cb143b85ee44793620237f94a714b01ba1c443bbf8b8377c32db7" exitCode=0
Oct 30 19:44:13.311805 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:13.311653    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"
Oct 30 19:44:13.311805 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:13.311682    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg" event={"ID":"7de8a961-4f64-4564-bb6c-d517bff87d28","Type":"ContainerDied","Data":"fcde31d7ee2cb143b85ee44793620237f94a714b01ba1c443bbf8b8377c32db7"}
Oct 30 19:44:13.311805 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:13.311730    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg" event={"ID":"7de8a961-4f64-4564-bb6c-d517bff87d28","Type":"ContainerDied","Data":"5b5c4a3fd85b49e16e1d14701a18aa3bb5e70bc7f4802d4fe878009bb141ebda"}
Oct 30 19:44:13.311805 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:13.311750    6030 scope.go:122] "RemoveContainer" containerID="fcde31d7ee2cb143b85ee44793620237f94a714b01ba1c443bbf8b8377c32db7"
Oct 30 19:44:13.313670 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:13.313646    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9" event={"ID":"8c76eb36-5c29-41fa-8289-d8f08fba2181","Type":"ContainerStarted","Data":"eccd86168122e976ae0bf9ee144d4b6bb0ce462315fafd25d506b52f7f8712af"}
Oct 30 19:44:13.313796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:13.313692    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9" event={"ID":"8c76eb36-5c29-41fa-8289-d8f08fba2181","Type":"ContainerStarted","Data":"5483220e66417f21cfc33d47d3996b9c112af615cdf1da2ce3b5ea82d3cef922"}
Oct 30 19:44:13.320944 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:13.320922    6030 scope.go:122] "RemoveContainer" containerID="fcde31d7ee2cb143b85ee44793620237f94a714b01ba1c443bbf8b8377c32db7"
Oct 30 19:44:13.321282 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:44:13.321262    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcde31d7ee2cb143b85ee44793620237f94a714b01ba1c443bbf8b8377c32db7\": container with ID starting with fcde31d7ee2cb143b85ee44793620237f94a714b01ba1c443bbf8b8377c32db7 not found: ID does not exist" containerID="fcde31d7ee2cb143b85ee44793620237f94a714b01ba1c443bbf8b8377c32db7"
Oct 30 19:44:13.321344 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:13.321292    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcde31d7ee2cb143b85ee44793620237f94a714b01ba1c443bbf8b8377c32db7"} err="failed to get container status \"fcde31d7ee2cb143b85ee44793620237f94a714b01ba1c443bbf8b8377c32db7\": rpc error: code = NotFound desc = could not find container \"fcde31d7ee2cb143b85ee44793620237f94a714b01ba1c443bbf8b8377c32db7\": container with ID starting with fcde31d7ee2cb143b85ee44793620237f94a714b01ba1c443bbf8b8377c32db7 not found: ID does not exist"
Oct 30 19:44:13.339924 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:13.339895    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"]
Oct 30 19:44:13.344900 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:13.344872    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/e2e-conflict-a-05cbc882-kserve-6f8cffb98b-6kdlg"]
Oct 30 19:44:13.382071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:13.380938    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9" podStartSLOduration=2.380913927 podStartE2EDuration="2.380913927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:13.377571906 +0000 UTC m=+732.293178763" watchObservedRunningTime="2026-10-30 19:44:13.380913927 +0000 UTC m=+732.296520800"
Oct 30 19:44:13.994709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:13.994667    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:13.996293 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:13.996268    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:44:13.996425 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:13.996322    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9" podUID="8c76eb36-5c29-41fa-8289-d8f08fba2181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:44:14.975080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:14.973590    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:44:14.975080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:14.973657    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9" podUID="8c76eb36-5c29-41fa-8289-d8f08fba2181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:44:15.237611 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:15.234870    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7de8a961-4f64-4564-bb6c-d517bff87d28" path="/var/lib/kubelet/pods/7de8a961-4f64-4564-bb6c-d517bff87d28/volumes"
Oct 30 19:44:15.983068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:15.980992    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:44:15.983068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:15.981084    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9" podUID="8c76eb36-5c29-41fa-8289-d8f08fba2181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:44:16.475071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.467497    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"]
Oct 30 19:44:16.508072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.505597    6030 cpu_manager.go:414] "containerMap: removing container" podUID="7de8a961-4f64-4564-bb6c-d517bff87d28" containerName="main"
Oct 30 19:44:16.508072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.505622    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de8a961-4f64-4564-bb6c-d517bff87d28" containerName="main"
Oct 30 19:44:16.508072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.505823    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de8a961-4f64-4564-bb6c-d517bff87d28" containerName="main"
Oct 30 19:44:16.556073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.555263    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"]
Oct 30 19:44:16.556073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.555402    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.700803 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.606953    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.700803 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.606997    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.700803 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.607071    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.700803 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.607096    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.700803 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.607118    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70cb2df1-ae6b-49d3-a989-db43421d61d6-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.700803 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.607179    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbqwl\" (UniqueName: \"kubernetes.io/projected/70cb2df1-ae6b-49d3-a989-db43421d61d6-kube-api-access-bbqwl\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.700803 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.607235    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.700803 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.607311    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.700803 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.607342    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.726117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.717706    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.726117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.717813    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.726117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.717852    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.726117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.717896    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.726117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.717932    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.726117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.717995    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.726117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.718020    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.726117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.718063    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70cb2df1-ae6b-49d3-a989-db43421d61d6-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.726117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.718141    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbqwl\" (UniqueName: \"kubernetes.io/projected/70cb2df1-ae6b-49d3-a989-db43421d61d6-kube-api-access-bbqwl\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.726117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.718947    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.780073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.727020    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.780073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.727394    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.780073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.727682    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.780073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.752134    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.780073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.753363    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70cb2df1-ae6b-49d3-a989-db43421d61d6-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.780073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.756551    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.798071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.786570    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.817104 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.800283    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:44:16.817104 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.801034    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.817104 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.804572    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbqwl\" (UniqueName: \"kubernetes.io/projected/70cb2df1-ae6b-49d3-a989-db43421d61d6-kube-api-access-bbqwl\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.817104 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.805069    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:16.862187 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.862142    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.863468 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.863437    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.898183 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.898134    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.962969 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.962929    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.970666 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.970629    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:44:16.970905 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.970881    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9" podUID="8c76eb36-5c29-41fa-8289-d8f08fba2181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:44:16.993539 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:16.993450    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:17.116072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:17.112164    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:44:17.626069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:17.625221    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:17.660071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:17.659251    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:17.763070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:17.759181    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:44:17.763070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:17.759609    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:44:17.763070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:17.761322    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:17.763070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:17.761626    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:44:17.804073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:17.803175    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:44:17.804073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:17.803613    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:44:17.866070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:17.865141    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:17.866070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:17.865552    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 19:44:17.894119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:17.893288    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:44:17.894119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:17.893359    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:44:17.919072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:17.902148    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 19:44:17.919072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:17.902548    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:44:18.049244 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:17.971457    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:18.049244 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:17.971861    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:44:18.204625 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:18.132497    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:44:18.204625 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:18.132557    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9" podUID="8c76eb36-5c29-41fa-8289-d8f08fba2181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:44:18.337905 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:18.337875    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:18.686071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:18.660945    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:18.700599 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:18.697952    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:44:18.700599 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:18.674129    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:18.734361 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:18.734320    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:18.738062 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:18.738000    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:44:18.738340 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:18.738313    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:44:18.738571 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:18.738551    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:44:18.906153 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:18.770600    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"]
Oct 30 19:44:19.173438 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:18.777191    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:19.173999 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:18.777239    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:19.195564 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:18.811180    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:19.196485 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:19.030143    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:19.215685 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:19.030848    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:19.216187 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:19.031086    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 19:44:19.216539 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:19.031110    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 19:44:19.216883 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:19.031170    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:44:19.217080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:19.217032    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9" podUID="8c76eb36-5c29-41fa-8289-d8f08fba2181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:44:19.217223 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:19.123154    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:19.354263 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:19.339108    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded"
Oct 30 19:44:19.388087 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:19.387295    6030 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:19.573615 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:19.573585    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:44:19.714120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:19.708508    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:19.789468 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:19.789421    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf" event={"ID":"70cb2df1-ae6b-49d3-a989-db43421d61d6","Type":"ContainerStarted","Data":"7aebe46041acbcd15dac2c2ad79d0e77d744abe9e5d992388284dc441aaf543d"}
Oct 30 19:44:20.062405 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.062315    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:44:20.062683 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.062654    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9" podUID="8c76eb36-5c29-41fa-8289-d8f08fba2181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:44:20.178983 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.178931    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:20.195472 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.195433    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:44:20.200066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.196457    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:44:20.221816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.221774    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:20.223526 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.223494    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:20.224082 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.223720    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:20.224909 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.223748    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 19:44:20.226069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.225532    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:44:20.315728 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.315636    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Oct 30 19:44:20.709211 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.709163    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:20.856473 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.855420    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf" event={"ID":"70cb2df1-ae6b-49d3-a989-db43421d61d6","Type":"ContainerStarted","Data":"ee9f01e23e5449306590c724d6eefe9bbf0c2c7849b56e70130d640b0cea6113"}
Oct 30 19:44:20.891699 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.891641    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf" podStartSLOduration=4.891621177 podStartE2EDuration="4.891621177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:20.889687595 +0000 UTC m=+739.805294451" watchObservedRunningTime="2026-10-30 19:44:20.891621177 +0000 UTC m=+739.807228034"
Oct 30 19:44:20.940174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.938835    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:44:20.944424 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.944397    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:44:20.949062 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.946725    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:44:20.967932 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.967262    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:44:20.971117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.970676    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:44:20.971117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.970738    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9" podUID="8c76eb36-5c29-41fa-8289-d8f08fba2181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:44:20.976727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:20.976673    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:44:21.019145 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:21.018990    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:21.023186 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:21.019919    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 19:44:21.023186 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:21.019989    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf" podUID="70cb2df1-ae6b-49d3-a989-db43421d61d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 19:44:21.023186 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:21.021496    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:44:21.149601 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:21.149570    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:44:21.160188 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:21.160160    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:44:22.002986 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.002945    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 19:44:22.006725 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.006684    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf" podUID="70cb2df1-ae6b-49d3-a989-db43421d61d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 19:44:22.044452 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.044340    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"]
Oct 30 19:44:22.050339 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.050302    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.097067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.089797    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.097067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.089902    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.097067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.089932    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.097067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.089964    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.097067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.090006    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.097067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.090175    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487f842d-ad13-45d8-8b55-fbf448ec511d-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.097067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.090222    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.097067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.090248    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgntr\" (UniqueName: \"kubernetes.io/projected/487f842d-ad13-45d8-8b55-fbf448ec511d-kube-api-access-jgntr\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.097067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.090276    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.119145 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.119104    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"]
Oct 30 19:44:22.203174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.194760    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487f842d-ad13-45d8-8b55-fbf448ec511d-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.203174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.194826    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.203174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.194855    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgntr\" (UniqueName: \"kubernetes.io/projected/487f842d-ad13-45d8-8b55-fbf448ec511d-kube-api-access-jgntr\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.203174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.194885    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.203174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.194963    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.203174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.195036    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.203174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.195084    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.203174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.195115    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.203174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.195157    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.203174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.196333    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487f842d-ad13-45d8-8b55-fbf448ec511d-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.203174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.196618    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.203174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.197118    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.203174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.197367    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.213489 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.207437    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.213489 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.210629    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.219853 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.219827    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.225233 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.225204    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.225775 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.225750    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgntr\" (UniqueName: \"kubernetes.io/projected/487f842d-ad13-45d8-8b55-fbf448ec511d-kube-api-access-jgntr\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.366325 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.366237    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:22.702365 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.701365    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"]
Oct 30 19:44:22.705857 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:44:22.705796    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487f842d_ad13_45d8_8b55_fbf448ec511d.slice/crio-3d87131ddace5ab54b134fb40c8a49bf13320a4606ee5986481fb46180248170 WatchSource:0}: Error finding container 3d87131ddace5ab54b134fb40c8a49bf13320a4606ee5986481fb46180248170: Status 404 returned error can't find the container with id 3d87131ddace5ab54b134fb40c8a49bf13320a4606ee5986481fb46180248170
Oct 30 19:44:22.708360 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.708327    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:44:22.708454 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.708395    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:44:22.708454 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.708442    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:44:22.902443 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.902389    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq" event={"ID":"487f842d-ad13-45d8-8b55-fbf448ec511d","Type":"ContainerStarted","Data":"f33d47337250393337f478216b15cc0b25421f3e812abd6dcf960721deae5dde"}
Oct 30 19:44:22.902736 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.902704    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq" event={"ID":"487f842d-ad13-45d8-8b55-fbf448ec511d","Type":"ContainerStarted","Data":"3d87131ddace5ab54b134fb40c8a49bf13320a4606ee5986481fb46180248170"}
Oct 30 19:44:22.938070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:22.930842    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq" podStartSLOduration=0.93082363 podStartE2EDuration="930.82363ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:22.929944668 +0000 UTC m=+741.845551518" watchObservedRunningTime="2026-10-30 19:44:22.93082363 +0000 UTC m=+741.846430487"
Oct 30 19:44:23.002630 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:23.002152    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 19:44:23.002630 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:23.002213    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf" podUID="70cb2df1-ae6b-49d3-a989-db43421d61d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 19:44:23.025486 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:23.025447    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:44:23.025928 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:23.025503    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9" podUID="8c76eb36-5c29-41fa-8289-d8f08fba2181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:23.366876 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:23.366791    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:23.368353 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:23.368325    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:44:23.368496 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:23.368385    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq" podUID="487f842d-ad13-45d8-8b55-fbf448ec511d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:44:23.481070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:23.479378    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:23.481070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:23.480502    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:23.491071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:23.487291    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:44:24.000069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:23.999453    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 19:44:24.000069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:23.999527    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf" podUID="70cb2df1-ae6b-49d3-a989-db43421d61d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 19:44:24.367133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:24.367030    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:44:24.367133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:24.367121    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq" podUID="487f842d-ad13-45d8-8b55-fbf448ec511d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:44:24.997067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:24.996454    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 19:44:24.997067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:24.996530    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf" podUID="70cb2df1-ae6b-49d3-a989-db43421d61d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 19:44:25.367329 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:25.367184    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:44:25.367895 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:25.367851    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq" podUID="487f842d-ad13-45d8-8b55-fbf448ec511d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:44:26.008607 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.008569    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 19:44:26.008879 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.008848    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf" podUID="70cb2df1-ae6b-49d3-a989-db43421d61d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 19:44:26.367078 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.366975    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:44:26.367380 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.367348    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq" podUID="487f842d-ad13-45d8-8b55-fbf448ec511d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:44:26.626334 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.626287    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"]
Oct 30 19:44:26.630593 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.630564    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"
Oct 30 19:44:26.635845 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.635818    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"]
Oct 30 19:44:26.637016 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.636995    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:26.637518 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.637499    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 19:44:26.637643 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.637625    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:44:26.637782 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.637560    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:44:26.718825 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.718778    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18142c9-b841-4aca-9480-ad43ab4a0625-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590ea3fb-tlczp\" (UID: \"a18142c9-b841-4aca-9480-ad43ab4a0625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"
Oct 30 19:44:26.719320 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.719295    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18142c9-b841-4aca-9480-ad43ab4a0625-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590ea3fb-tlczp\" (UID: \"a18142c9-b841-4aca-9480-ad43ab4a0625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"
Oct 30 19:44:26.719655 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.719634    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-488bj\" (UniqueName: \"kubernetes.io/projected/a18142c9-b841-4aca-9480-ad43ab4a0625-kube-api-access-488bj\") pod \"maas-api-revoke-sub-590ea3fb-tlczp\" (UID: \"a18142c9-b841-4aca-9480-ad43ab4a0625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"
Oct 30 19:44:26.821447 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.821413    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18142c9-b841-4aca-9480-ad43ab4a0625-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590ea3fb-tlczp\" (UID: \"a18142c9-b841-4aca-9480-ad43ab4a0625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"
Oct 30 19:44:26.821613 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.821497    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18142c9-b841-4aca-9480-ad43ab4a0625-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590ea3fb-tlczp\" (UID: \"a18142c9-b841-4aca-9480-ad43ab4a0625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"
Oct 30 19:44:26.821613 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.821523    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-488bj\" (UniqueName: \"kubernetes.io/projected/a18142c9-b841-4aca-9480-ad43ab4a0625-kube-api-access-488bj\") pod \"maas-api-revoke-sub-590ea3fb-tlczp\" (UID: \"a18142c9-b841-4aca-9480-ad43ab4a0625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"
Oct 30 19:44:26.822594 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.822559    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18142c9-b841-4aca-9480-ad43ab4a0625-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590ea3fb-tlczp\" (UID: \"a18142c9-b841-4aca-9480-ad43ab4a0625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"
Oct 30 19:44:26.823949 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.823919    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18142c9-b841-4aca-9480-ad43ab4a0625-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590ea3fb-tlczp\" (UID: \"a18142c9-b841-4aca-9480-ad43ab4a0625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"
Oct 30 19:44:26.836310 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.836283    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-488bj\" (UniqueName: \"kubernetes.io/projected/a18142c9-b841-4aca-9480-ad43ab4a0625-kube-api-access-488bj\") pod \"maas-api-revoke-sub-590ea3fb-tlczp\" (UID: \"a18142c9-b841-4aca-9480-ad43ab4a0625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"
Oct 30 19:44:26.951114 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.951024    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"
Oct 30 19:44:26.995068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.994674    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 19:44:26.995068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:26.994735    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf" podUID="70cb2df1-ae6b-49d3-a989-db43421d61d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 19:44:27.259193 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:27.259162    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"]
Oct 30 19:44:27.268216 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:44:27.268186    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18142c9_b841_4aca_9480_ad43ab4a0625.slice/crio-c9ae93649139b3e07658385f70adae745d8ee466dc66f4b16bf8536ab9abb6a1 WatchSource:0}: Error finding container c9ae93649139b3e07658385f70adae745d8ee466dc66f4b16bf8536ab9abb6a1: Status 404 returned error can't find the container with id c9ae93649139b3e07658385f70adae745d8ee466dc66f4b16bf8536ab9abb6a1
Oct 30 19:44:27.367257 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:27.367222    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:44:27.367386 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:27.367274    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq" podUID="487f842d-ad13-45d8-8b55-fbf448ec511d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:44:27.948555 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:27.948459    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp" event={"ID":"a18142c9-b841-4aca-9480-ad43ab4a0625","Type":"ContainerStarted","Data":"1eb8494f85ead8143d271061b9c1af5d585889cd25be4091f5eb66c29e7d8fb6"}
Oct 30 19:44:27.948555 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:27.948523    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp" event={"ID":"a18142c9-b841-4aca-9480-ad43ab4a0625","Type":"ContainerStarted","Data":"c9ae93649139b3e07658385f70adae745d8ee466dc66f4b16bf8536ab9abb6a1"}
Oct 30 19:44:27.974072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:27.971786    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp" podStartSLOduration=1.971765078 podStartE2EDuration="1.971765078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:27.968234587 +0000 UTC m=+746.883841444" watchObservedRunningTime="2026-10-30 19:44:27.971765078 +0000 UTC m=+746.887371935"
Oct 30 19:44:28.219634 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:28.219538    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:28.241393 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:28.241369    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:28.366615 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:28.366577    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:44:28.366781 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:28.366668    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq" podUID="487f842d-ad13-45d8-8b55-fbf448ec511d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:44:28.457231 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:28.457166    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:28.458390 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:28.458358    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:28.459578 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:28.459560    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 19:44:28.956163 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:28.956125    6030 generic.go:356] "Generic (PLEG): container finished" podID="a18142c9-b841-4aca-9480-ad43ab4a0625" containerID="1eb8494f85ead8143d271061b9c1af5d585889cd25be4091f5eb66c29e7d8fb6" exitCode=0
Oct 30 19:44:28.956363 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:28.956213    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp" event={"ID":"a18142c9-b841-4aca-9480-ad43ab4a0625","Type":"ContainerDied","Data":"1eb8494f85ead8143d271061b9c1af5d585889cd25be4091f5eb66c29e7d8fb6"}
Oct 30 19:44:29.367221 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:29.367134    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:44:29.367221 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:29.367203    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq" podUID="487f842d-ad13-45d8-8b55-fbf448ec511d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:44:30.121254 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:30.121230    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"
Oct 30 19:44:30.174663 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:30.174630    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a18142c9-b841-4aca-9480-ad43ab4a0625-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18142c9-b841-4aca-9480-ad43ab4a0625-maas-api-xks-ca\") pod \"a18142c9-b841-4aca-9480-ad43ab4a0625\" (UID: \"a18142c9-b841-4aca-9480-ad43ab4a0625\") "
Oct 30 19:44:30.174852 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:30.174754    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a18142c9-b841-4aca-9480-ad43ab4a0625-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18142c9-b841-4aca-9480-ad43ab4a0625-maas-api-openshift-ca\") pod \"a18142c9-b841-4aca-9480-ad43ab4a0625\" (UID: \"a18142c9-b841-4aca-9480-ad43ab4a0625\") "
Oct 30 19:44:30.174852 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:30.174825    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18142c9-b841-4aca-9480-ad43ab4a0625-kube-api-access-488bj\" (UniqueName: \"kubernetes.io/projected/a18142c9-b841-4aca-9480-ad43ab4a0625-kube-api-access-488bj\") pod \"a18142c9-b841-4aca-9480-ad43ab4a0625\" (UID: \"a18142c9-b841-4aca-9480-ad43ab4a0625\") "
Oct 30 19:44:30.175186 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:30.175157    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18142c9-b841-4aca-9480-ad43ab4a0625-maas-api-openshift-ca" pod "a18142c9-b841-4aca-9480-ad43ab4a0625" (UID: "a18142c9-b841-4aca-9480-ad43ab4a0625"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:30.175397 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:30.175375    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18142c9-b841-4aca-9480-ad43ab4a0625-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:30.177239 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:30.177216    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18142c9-b841-4aca-9480-ad43ab4a0625-kube-api-access-488bj" pod "a18142c9-b841-4aca-9480-ad43ab4a0625" (UID: "a18142c9-b841-4aca-9480-ad43ab4a0625"). InnerVolumeSpecName "kube-api-access-488bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:30.177785 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:30.177767    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a18142c9-b841-4aca-9480-ad43ab4a0625-maas-api-xks-ca" pod "a18142c9-b841-4aca-9480-ad43ab4a0625" (UID: "a18142c9-b841-4aca-9480-ad43ab4a0625"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:30.276097 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:30.276005    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-488bj\" (UniqueName: \"kubernetes.io/projected/a18142c9-b841-4aca-9480-ad43ab4a0625-kube-api-access-488bj\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:30.276097 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:30.276036    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18142c9-b841-4aca-9480-ad43ab4a0625-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:30.438650 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:30.438617    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:30.439079 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:30.439019    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:30.439815 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:30.439793    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 19:44:30.969542 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:30.969514    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"
Oct 30 19:44:30.969713 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:30.969541    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp" event={"ID":"a18142c9-b841-4aca-9480-ad43ab4a0625","Type":"ContainerDied","Data":"c9ae93649139b3e07658385f70adae745d8ee466dc66f4b16bf8536ab9abb6a1"}
Oct 30 19:44:30.969713 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:30.969592    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9ae93649139b3e07658385f70adae745d8ee466dc66f4b16bf8536ab9abb6a1"
Oct 30 19:44:34.289892 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:44:34.289856    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:44:37.068941 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.068908    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"]
Oct 30 19:44:37.069521 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.069503    6030 cpu_manager.go:414] "containerMap: removing container" podUID="a18142c9-b841-4aca-9480-ad43ab4a0625" containerName="revoke-keys"
Oct 30 19:44:37.069568 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.069533    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18142c9-b841-4aca-9480-ad43ab4a0625" containerName="revoke-keys"
Oct 30 19:44:37.069678 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.069669    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18142c9-b841-4aca-9480-ad43ab4a0625" containerName="revoke-keys"
Oct 30 19:44:37.073120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.073104    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.085729 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.085704    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"]
Oct 30 19:44:37.137645 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.137606    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.137794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.137739    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.137868 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.137821    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2vvk\" (UniqueName: \"kubernetes.io/projected/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-kube-api-access-j2vvk\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.137868 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.137861    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.137993 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.137971    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.138038 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.138001    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.138123 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.138107    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.138158 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.138140    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.138201 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.138171    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.240254 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.240218    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.240254 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.240258    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.240509 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.240280    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.240509 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.240381    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.240509 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.240454    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.240509 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.240480    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2vvk\" (UniqueName: \"kubernetes.io/projected/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-kube-api-access-j2vvk\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.240509 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.240503    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.240752 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.240588    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.240752 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.240613    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.241014 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.240988    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.241112 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.241061    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.241181 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.241079    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.241332 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.241310    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.241406 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.241382    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.243384 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.243358    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.243584 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.243560    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.254485 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.254453    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.254830 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.254808    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2vvk\" (UniqueName: \"kubernetes.io/projected/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-kube-api-access-j2vvk\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.388938 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.388897    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:37.677552 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.677522    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"]
Oct 30 19:44:37.680002 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:44:37.679972    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2af060_ec91_4d0f_b61d_cfec82fc4e2f.slice/crio-68505c52f8038037234ab36dd723a8ff216601fea27866b63c748e3b41c1e01a WatchSource:0}: Error finding container 68505c52f8038037234ab36dd723a8ff216601fea27866b63c748e3b41c1e01a: Status 404 returned error can't find the container with id 68505c52f8038037234ab36dd723a8ff216601fea27866b63c748e3b41c1e01a
Oct 30 19:44:37.682597 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.682559    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:44:37.682704 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.682645    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:44:37.682704 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:37.682691    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:44:38.042881 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:38.042782    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn" event={"ID":"af2af060-ec91-4d0f-b61d-cfec82fc4e2f","Type":"ContainerStarted","Data":"fbfc3759e8d67a9ce6e8f418ea9a5c48c4309dc028fd9df23595308dd17fc1a1"}
Oct 30 19:44:38.042881 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:38.042855    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn" event={"ID":"af2af060-ec91-4d0f-b61d-cfec82fc4e2f","Type":"ContainerStarted","Data":"68505c52f8038037234ab36dd723a8ff216601fea27866b63c748e3b41c1e01a"}
Oct 30 19:44:38.072692 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:38.072625    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn" podStartSLOduration=1.072606292 podStartE2EDuration="1.072606292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:38.07024224 +0000 UTC m=+756.985849122" watchObservedRunningTime="2026-10-30 19:44:38.072606292 +0000 UTC m=+756.988213153"
Oct 30 19:44:38.389813 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:38.389782    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:38.391671 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:38.391644    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:44:38.391805 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:38.391690    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn" podUID="af2af060-ec91-4d0f-b61d-cfec82fc4e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:44:39.390070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:39.389437    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:44:39.390070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:39.389504    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn" podUID="af2af060-ec91-4d0f-b61d-cfec82fc4e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:44:40.391460 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:40.391420    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:44:40.392101 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:40.392065    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn" podUID="af2af060-ec91-4d0f-b61d-cfec82fc4e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:44:41.391465 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:41.391424    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:44:41.392030 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:41.391996    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn" podUID="af2af060-ec91-4d0f-b61d-cfec82fc4e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:44:42.391349 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:42.391311    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:44:42.391657 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:42.391617    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn" podUID="af2af060-ec91-4d0f-b61d-cfec82fc4e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:44:43.392070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:43.391439    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:44:43.392070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:43.391505    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn" podUID="af2af060-ec91-4d0f-b61d-cfec82fc4e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:44:44.390140 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:44.390105    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:44:44.390332 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:44.390173    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn" podUID="af2af060-ec91-4d0f-b61d-cfec82fc4e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:44:46.425926 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:46.425883    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:44:46.426362 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:46.425990    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn" podUID="af2af060-ec91-4d0f-b61d-cfec82fc4e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:46.462066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:46.462005    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:46.462799 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:46.462775    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:46.464168 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:46.464152    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 19:44:48.671206 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:48.671170    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"]
Oct 30 19:44:48.675265 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:48.675239    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"
Oct 30 19:44:48.681442 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:48.681411    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:44:48.681602 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:48.681523    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:44:48.681602 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:48.681590    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 19:44:48.681734 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:48.681606    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:48.683489 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:48.683464    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"]
Oct 30 19:44:48.804280 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:48.804239    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fbd5646-a264-465a-8159-2e01c0526294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eae23df-jbqgc\" (UID: \"1fbd5646-a264-465a-8159-2e01c0526294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"
Oct 30 19:44:48.804531 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:48.804507    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fbd5646-a264-465a-8159-2e01c0526294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eae23df-jbqgc\" (UID: \"1fbd5646-a264-465a-8159-2e01c0526294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"
Oct 30 19:44:48.804632 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:48.804610    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc8cp\" (UniqueName: \"kubernetes.io/projected/1fbd5646-a264-465a-8159-2e01c0526294-kube-api-access-mc8cp\") pod \"maas-api-revoke-sub-8eae23df-jbqgc\" (UID: \"1fbd5646-a264-465a-8159-2e01c0526294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"
Oct 30 19:44:48.906021 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:48.905982    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fbd5646-a264-465a-8159-2e01c0526294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eae23df-jbqgc\" (UID: \"1fbd5646-a264-465a-8159-2e01c0526294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"
Oct 30 19:44:48.906215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:48.906084    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc8cp\" (UniqueName: \"kubernetes.io/projected/1fbd5646-a264-465a-8159-2e01c0526294-kube-api-access-mc8cp\") pod \"maas-api-revoke-sub-8eae23df-jbqgc\" (UID: \"1fbd5646-a264-465a-8159-2e01c0526294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"
Oct 30 19:44:48.906215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:48.906120    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fbd5646-a264-465a-8159-2e01c0526294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eae23df-jbqgc\" (UID: \"1fbd5646-a264-465a-8159-2e01c0526294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"
Oct 30 19:44:48.906890 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:48.906860    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fbd5646-a264-465a-8159-2e01c0526294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eae23df-jbqgc\" (UID: \"1fbd5646-a264-465a-8159-2e01c0526294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"
Oct 30 19:44:48.908561 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:48.908534    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fbd5646-a264-465a-8159-2e01c0526294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eae23df-jbqgc\" (UID: \"1fbd5646-a264-465a-8159-2e01c0526294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"
Oct 30 19:44:48.920636 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:48.920602    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc8cp\" (UniqueName: \"kubernetes.io/projected/1fbd5646-a264-465a-8159-2e01c0526294-kube-api-access-mc8cp\") pod \"maas-api-revoke-sub-8eae23df-jbqgc\" (UID: \"1fbd5646-a264-465a-8159-2e01c0526294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"
Oct 30 19:44:48.993285 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:48.993195    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"
Oct 30 19:44:49.295389 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:49.295366    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"]
Oct 30 19:44:50.166664 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:50.166604    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc" event={"ID":"1fbd5646-a264-465a-8159-2e01c0526294","Type":"ContainerStarted","Data":"7da3188965c5419129016e5b6865f6a6619d471b731d7156edc7504a3d8ee946"}
Oct 30 19:44:50.167348 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:50.167289    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc" event={"ID":"1fbd5646-a264-465a-8159-2e01c0526294","Type":"ContainerStarted","Data":"506478118f01b3e8e16a0f5cf09e6276cda125ac53ed177d18b77bb128de5c5e"}
Oct 30 19:44:51.230291 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:51.230100    6030 generic.go:356] "Generic (PLEG): container finished" podID="1fbd5646-a264-465a-8159-2e01c0526294" containerID="7da3188965c5419129016e5b6865f6a6619d471b731d7156edc7504a3d8ee946" exitCode=0
Oct 30 19:44:51.241547 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:51.241501    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc" event={"ID":"1fbd5646-a264-465a-8159-2e01c0526294","Type":"ContainerDied","Data":"7da3188965c5419129016e5b6865f6a6619d471b731d7156edc7504a3d8ee946"}
Oct 30 19:44:52.460509 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:52.460480    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"
Oct 30 19:44:52.626119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:52.622823    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fbd5646-a264-465a-8159-2e01c0526294-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fbd5646-a264-465a-8159-2e01c0526294-maas-api-openshift-ca\") pod \"1fbd5646-a264-465a-8159-2e01c0526294\" (UID: \"1fbd5646-a264-465a-8159-2e01c0526294\") "
Oct 30 19:44:52.626119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:52.622916    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fbd5646-a264-465a-8159-2e01c0526294-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fbd5646-a264-465a-8159-2e01c0526294-maas-api-xks-ca\") pod \"1fbd5646-a264-465a-8159-2e01c0526294\" (UID: \"1fbd5646-a264-465a-8159-2e01c0526294\") "
Oct 30 19:44:52.626119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:52.622949    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fbd5646-a264-465a-8159-2e01c0526294-kube-api-access-mc8cp\" (UniqueName: \"kubernetes.io/projected/1fbd5646-a264-465a-8159-2e01c0526294-kube-api-access-mc8cp\") pod \"1fbd5646-a264-465a-8159-2e01c0526294\" (UID: \"1fbd5646-a264-465a-8159-2e01c0526294\") "
Oct 30 19:44:52.638565 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:52.638522    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fbd5646-a264-465a-8159-2e01c0526294-maas-api-openshift-ca" pod "1fbd5646-a264-465a-8159-2e01c0526294" (UID: "1fbd5646-a264-465a-8159-2e01c0526294"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:52.671492 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:52.664738    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fbd5646-a264-465a-8159-2e01c0526294-kube-api-access-mc8cp" pod "1fbd5646-a264-465a-8159-2e01c0526294" (UID: "1fbd5646-a264-465a-8159-2e01c0526294"). InnerVolumeSpecName "kube-api-access-mc8cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:52.671492 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:52.664831    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fbd5646-a264-465a-8159-2e01c0526294-maas-api-xks-ca" pod "1fbd5646-a264-465a-8159-2e01c0526294" (UID: "1fbd5646-a264-465a-8159-2e01c0526294"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:52.725240 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:52.725193    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fbd5646-a264-465a-8159-2e01c0526294-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:52.725240 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:52.725236    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fbd5646-a264-465a-8159-2e01c0526294-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:52.725477 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:52.725253    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc8cp\" (UniqueName: \"kubernetes.io/projected/1fbd5646-a264-465a-8159-2e01c0526294-kube-api-access-mc8cp\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.274083 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:53.271913    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc" event={"ID":"1fbd5646-a264-465a-8159-2e01c0526294","Type":"ContainerDied","Data":"506478118f01b3e8e16a0f5cf09e6276cda125ac53ed177d18b77bb128de5c5e"}
Oct 30 19:44:53.274083 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:53.271980    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="506478118f01b3e8e16a0f5cf09e6276cda125ac53ed177d18b77bb128de5c5e"
Oct 30 19:44:53.274083 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:53.273262    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"
Oct 30 19:44:54.155497 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:54.155381    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:54.209213 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:54.209172    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:54.222988 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:54.222948    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:54.223539 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:54.223221    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 19:44:54.224283 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:54.223399    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:54.233604 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:54.233566    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:54.265733 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:54.265205    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:55.181924 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:55.181882    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:55.234845 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:55.233178    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:55.234845 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:55.233608    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 19:44:55.234845 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:55.233904    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 19:44:55.234845 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:55.234530    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:44:55.260071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:55.247327    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:55.260071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:55.247394    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:44:55.260071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:55.247430    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:44:55.260071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:55.247464    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:44:55.283073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:55.272239    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:55.283073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:55.272901    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:44:55.754073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:55.711319    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:55.998075 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:55.993541    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:56.023653 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:56.003233    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:44:56.220121 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:56.172682    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:44:56.220121 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:56.173506    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:56.220121 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:56.183202    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:56.220121 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:56.201297    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:44:56.251334 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:56.251290    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:56.319844 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:56.267532    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:56.319844 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:56.283689    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:44:56.319844 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:56.283730    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 19:44:56.319844 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:56.284502    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:56.319844 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:56.284547    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 19:44:56.793242 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:56.793210    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:44:56.812443 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:56.812407    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:44:56.812741 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:56.812714    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5" podUID="be081d10-b3c7-4a5a-9abf-9989ca04e364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:56.841743 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:56.840868    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:44:56.970066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:56.969147    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:44:57.004475 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:57.004408    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:44:57.004749 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:57.004725    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz" podUID="2c441eb5-f8ed-413a-b77c-4a8e043c431e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:57.039292 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:57.039262    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:44:57.051404 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:44:57.051322    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:45:00.226147 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.226110    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t"]
Oct 30 19:45:00.274137 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.270826    6030 cpu_manager.go:414] "containerMap: removing container" podUID="1fbd5646-a264-465a-8159-2e01c0526294" containerName="revoke-keys"
Oct 30 19:45:00.274137 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.270862    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fbd5646-a264-465a-8159-2e01c0526294" containerName="revoke-keys"
Oct 30 19:45:00.274137 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.271031    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fbd5646-a264-465a-8159-2e01c0526294" containerName="revoke-keys"
Oct 30 19:45:00.332731 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.315440    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t"
Oct 30 19:45:00.341268 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.338940    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9"]
Oct 30 19:45:00.354551 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.354499    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf"]
Oct 30 19:45:00.355142 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.355118    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 19:45:00.356415 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.356394    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:45:00.356738 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.356726    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9"
Oct 30 19:45:00.366600 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.359130    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:45:00.368839 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.368816    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t"]
Oct 30 19:45:00.368958 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.368848    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9"]
Oct 30 19:45:00.368958 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.368865    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf"]
Oct 30 19:45:00.368958 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.368882    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt"]
Oct 30 19:45:00.369267 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.369244    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf"
Oct 30 19:45:00.372281 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.372264    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt"]
Oct 30 19:45:00.372359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.372336    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt"
Oct 30 19:45:00.402095 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.402064    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pdtc\" (UniqueName: \"kubernetes.io/projected/d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2-kube-api-access-5pdtc\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t\" (UID: \"d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t"
Oct 30 19:45:00.402325 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.402305    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a853a9-7fc4-4537-8982-ff5943631c6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9\" (UID: \"00a853a9-7fc4-4537-8982-ff5943631c6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9"
Oct 30 19:45:00.402456 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.402442    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t\" (UID: \"d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t"
Oct 30 19:45:00.402596 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.402581    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce8f341-0efa-49c8-96aa-27772a49670b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-9hgvt\" (UID: \"0ce8f341-0efa-49c8-96aa-27772a49670b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt"
Oct 30 19:45:00.402755 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.402723    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2895p\" (UniqueName: \"kubernetes.io/projected/d32cf0e1-c0c1-4cbc-98ad-bd177dff4711-kube-api-access-2895p\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf\" (UID: \"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf"
Oct 30 19:45:00.402852 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.402769    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32cf0e1-c0c1-4cbc-98ad-bd177dff4711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf\" (UID: \"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf"
Oct 30 19:45:00.402892 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.402860    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj4zs\" (UniqueName: \"kubernetes.io/projected/0ce8f341-0efa-49c8-96aa-27772a49670b-kube-api-access-nj4zs\") pod \"maas-api-key-cleanup-29889825-9hgvt\" (UID: \"0ce8f341-0efa-49c8-96aa-27772a49670b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt"
Oct 30 19:45:00.402947 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.402929    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7lnr\" (UniqueName: \"kubernetes.io/projected/00a853a9-7fc4-4537-8982-ff5943631c6c-kube-api-access-c7lnr\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9\" (UID: \"00a853a9-7fc4-4537-8982-ff5943631c6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9"
Oct 30 19:45:00.504813 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.504704    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pdtc\" (UniqueName: \"kubernetes.io/projected/d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2-kube-api-access-5pdtc\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t\" (UID: \"d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t"
Oct 30 19:45:00.504813 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.504774    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a853a9-7fc4-4537-8982-ff5943631c6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9\" (UID: \"00a853a9-7fc4-4537-8982-ff5943631c6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9"
Oct 30 19:45:00.504813 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.504810    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t\" (UID: \"d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t"
Oct 30 19:45:00.505139 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.504857    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce8f341-0efa-49c8-96aa-27772a49670b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-9hgvt\" (UID: \"0ce8f341-0efa-49c8-96aa-27772a49670b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt"
Oct 30 19:45:00.505139 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.504921    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2895p\" (UniqueName: \"kubernetes.io/projected/d32cf0e1-c0c1-4cbc-98ad-bd177dff4711-kube-api-access-2895p\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf\" (UID: \"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf"
Oct 30 19:45:00.505139 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.504964    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32cf0e1-c0c1-4cbc-98ad-bd177dff4711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf\" (UID: \"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf"
Oct 30 19:45:00.505139 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.505014    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj4zs\" (UniqueName: \"kubernetes.io/projected/0ce8f341-0efa-49c8-96aa-27772a49670b-kube-api-access-nj4zs\") pod \"maas-api-key-cleanup-29889825-9hgvt\" (UID: \"0ce8f341-0efa-49c8-96aa-27772a49670b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt"
Oct 30 19:45:00.505139 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.505088    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7lnr\" (UniqueName: \"kubernetes.io/projected/00a853a9-7fc4-4537-8982-ff5943631c6c-kube-api-access-c7lnr\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9\" (UID: \"00a853a9-7fc4-4537-8982-ff5943631c6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9"
Oct 30 19:45:00.506943 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.506912    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a853a9-7fc4-4537-8982-ff5943631c6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9\" (UID: \"00a853a9-7fc4-4537-8982-ff5943631c6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9"
Oct 30 19:45:00.507336 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.507311    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t\" (UID: \"d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t"
Oct 30 19:45:00.507823 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.507788    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce8f341-0efa-49c8-96aa-27772a49670b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-9hgvt\" (UID: \"0ce8f341-0efa-49c8-96aa-27772a49670b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt"
Oct 30 19:45:00.508021 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.507999    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32cf0e1-c0c1-4cbc-98ad-bd177dff4711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf\" (UID: \"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf"
Oct 30 19:45:00.568843 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.564817    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7lnr\" (UniqueName: \"kubernetes.io/projected/00a853a9-7fc4-4537-8982-ff5943631c6c-kube-api-access-c7lnr\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9\" (UID: \"00a853a9-7fc4-4537-8982-ff5943631c6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9"
Oct 30 19:45:00.568843 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.567605    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj4zs\" (UniqueName: \"kubernetes.io/projected/0ce8f341-0efa-49c8-96aa-27772a49670b-kube-api-access-nj4zs\") pod \"maas-api-key-cleanup-29889825-9hgvt\" (UID: \"0ce8f341-0efa-49c8-96aa-27772a49670b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt"
Oct 30 19:45:00.606070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.600891    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2895p\" (UniqueName: \"kubernetes.io/projected/d32cf0e1-c0c1-4cbc-98ad-bd177dff4711-kube-api-access-2895p\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf\" (UID: \"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf"
Oct 30 19:45:00.606070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.605663    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pdtc\" (UniqueName: \"kubernetes.io/projected/d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2-kube-api-access-5pdtc\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t\" (UID: \"d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t"
Oct 30 19:45:00.679070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.678253    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t"
Oct 30 19:45:00.707272 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.707238    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9"
Oct 30 19:45:00.714907 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.714877    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf"
Oct 30 19:45:00.720013 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.719990    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt"
Oct 30 19:45:00.983433 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.983403    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"]
Oct 30 19:45:00.988683 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.988657    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"
Oct 30 19:45:00.997889 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.997787    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:00.997889 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:00.997828    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t"]
Oct 30 19:45:01.000204 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.000179    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"]
Oct 30 19:45:01.013802 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.013744    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a625e6-791a-4492-9cb5-621e5c52f41a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75f67ec5-rc9z6\" (UID: \"a9a625e6-791a-4492-9cb5-621e5c52f41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"
Oct 30 19:45:01.014024 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.013962    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a625e6-791a-4492-9cb5-621e5c52f41a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75f67ec5-rc9z6\" (UID: \"a9a625e6-791a-4492-9cb5-621e5c52f41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"
Oct 30 19:45:01.014220 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.014201    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckvj7\" (UniqueName: \"kubernetes.io/projected/a9a625e6-791a-4492-9cb5-621e5c52f41a-kube-api-access-ckvj7\") pod \"maas-api-revoke-sub-75f67ec5-rc9z6\" (UID: \"a9a625e6-791a-4492-9cb5-621e5c52f41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"
Oct 30 19:45:01.015616 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.015597    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9"]
Oct 30 19:45:01.021145 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:45:01.021116    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a853a9_7fc4_4537_8982_ff5943631c6c.slice/crio-a59f5addf18330fa2dbc0acd7d22a5d6cef8e46a9e55d64631882d8bd4d7b5ef WatchSource:0}: Error finding container a59f5addf18330fa2dbc0acd7d22a5d6cef8e46a9e55d64631882d8bd4d7b5ef: Status 404 returned error can't find the container with id a59f5addf18330fa2dbc0acd7d22a5d6cef8e46a9e55d64631882d8bd4d7b5ef
Oct 30 19:45:01.021663 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:45:01.021628    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32cf0e1_c0c1_4cbc_98ad_bd177dff4711.slice/crio-c86447df1190d3ace58ce920632d351569c561f8d2cf90a00d649f88c9daa351 WatchSource:0}: Error finding container c86447df1190d3ace58ce920632d351569c561f8d2cf90a00d649f88c9daa351: Status 404 returned error can't find the container with id c86447df1190d3ace58ce920632d351569c561f8d2cf90a00d649f88c9daa351
Oct 30 19:45:01.021814 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.021792    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf"]
Oct 30 19:45:01.035080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.035023    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt"]
Oct 30 19:45:01.043081 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:45:01.043027    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce8f341_0efa_49c8_96aa_27772a49670b.slice/crio-100f0989f09f20aaa3435d2806f2575f5222edc8215b53cdcadfb322479e1de7 WatchSource:0}: Error finding container 100f0989f09f20aaa3435d2806f2575f5222edc8215b53cdcadfb322479e1de7: Status 404 returned error can't find the container with id 100f0989f09f20aaa3435d2806f2575f5222edc8215b53cdcadfb322479e1de7
Oct 30 19:45:01.115064 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.115013    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckvj7\" (UniqueName: \"kubernetes.io/projected/a9a625e6-791a-4492-9cb5-621e5c52f41a-kube-api-access-ckvj7\") pod \"maas-api-revoke-sub-75f67ec5-rc9z6\" (UID: \"a9a625e6-791a-4492-9cb5-621e5c52f41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"
Oct 30 19:45:01.115252 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.115229    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a625e6-791a-4492-9cb5-621e5c52f41a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75f67ec5-rc9z6\" (UID: \"a9a625e6-791a-4492-9cb5-621e5c52f41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"
Oct 30 19:45:01.115337 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.115254    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a625e6-791a-4492-9cb5-621e5c52f41a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75f67ec5-rc9z6\" (UID: \"a9a625e6-791a-4492-9cb5-621e5c52f41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"
Oct 30 19:45:01.115923 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.115898    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a625e6-791a-4492-9cb5-621e5c52f41a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75f67ec5-rc9z6\" (UID: \"a9a625e6-791a-4492-9cb5-621e5c52f41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"
Oct 30 19:45:01.179086 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.178453    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a625e6-791a-4492-9cb5-621e5c52f41a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75f67ec5-rc9z6\" (UID: \"a9a625e6-791a-4492-9cb5-621e5c52f41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"
Oct 30 19:45:01.180718 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.180688    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckvj7\" (UniqueName: \"kubernetes.io/projected/a9a625e6-791a-4492-9cb5-621e5c52f41a-kube-api-access-ckvj7\") pod \"maas-api-revoke-sub-75f67ec5-rc9z6\" (UID: \"a9a625e6-791a-4492-9cb5-621e5c52f41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"
Oct 30 19:45:01.316538 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.316502    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"
Oct 30 19:45:01.602685 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.602657    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"]
Oct 30 19:45:01.604065 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:45:01.604018    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a625e6_791a_4492_9cb5_621e5c52f41a.slice/crio-3b9e02585080a111e51d4ad70c85019b1f53ee68468ef6ec4203394601e56f9a WatchSource:0}: Error finding container 3b9e02585080a111e51d4ad70c85019b1f53ee68468ef6ec4203394601e56f9a: Status 404 returned error can't find the container with id 3b9e02585080a111e51d4ad70c85019b1f53ee68468ef6ec4203394601e56f9a
Oct 30 19:45:01.614624 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.614588    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9" event={"ID":"00a853a9-7fc4-4537-8982-ff5943631c6c","Type":"ContainerStarted","Data":"95b95f8fe50ef4c9273ccbaa1739ef007ea2af67684c982bc369e72eec75fc88"}
Oct 30 19:45:01.614732 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.614660    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9" event={"ID":"00a853a9-7fc4-4537-8982-ff5943631c6c","Type":"ContainerStarted","Data":"a59f5addf18330fa2dbc0acd7d22a5d6cef8e46a9e55d64631882d8bd4d7b5ef"}
Oct 30 19:45:01.616364 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.616340    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt" event={"ID":"0ce8f341-0efa-49c8-96aa-27772a49670b","Type":"ContainerStarted","Data":"1471c1a55da29f3faa032be477a7c9b6c123faf98a0535b2c9fc62826dbf6fe0"}
Oct 30 19:45:01.616489 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.616392    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt" event={"ID":"0ce8f341-0efa-49c8-96aa-27772a49670b","Type":"ContainerStarted","Data":"100f0989f09f20aaa3435d2806f2575f5222edc8215b53cdcadfb322479e1de7"}
Oct 30 19:45:01.618260 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.618231    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t" event={"ID":"d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2","Type":"ContainerStarted","Data":"a82d5fed19c3f153335b389f732919283deada290479d73b2488a543f89825ab"}
Oct 30 19:45:01.618358 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.618276    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t" event={"ID":"d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2","Type":"ContainerStarted","Data":"17444ab4b5a0ebf868184848d68ba572abc52658ddd9741e7407442b8746f44e"}
Oct 30 19:45:01.620097 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.620072    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf" event={"ID":"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711","Type":"ContainerStarted","Data":"38899f5fbf80244ec3922b5ffcaef288f0ebf98f0863912b257111ec1bb83bb1"}
Oct 30 19:45:01.620202 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.620122    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf" event={"ID":"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711","Type":"ContainerStarted","Data":"c86447df1190d3ace58ce920632d351569c561f8d2cf90a00d649f88c9daa351"}
Oct 30 19:45:01.621408 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.621383    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6" event={"ID":"a9a625e6-791a-4492-9cb5-621e5c52f41a","Type":"ContainerStarted","Data":"3b9e02585080a111e51d4ad70c85019b1f53ee68468ef6ec4203394601e56f9a"}
Oct 30 19:45:01.636827 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.636786    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9" podStartSLOduration=1.636773828 podStartE2EDuration="1.636773828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:01.63397079 +0000 UTC m=+780.549577647" watchObservedRunningTime="2026-10-30 19:45:01.636773828 +0000 UTC m=+780.552380683"
Oct 30 19:45:01.652923 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.652872    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt" podStartSLOduration=1.6528595259999999 podStartE2EDuration="1.652859526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:01.652219627 +0000 UTC m=+780.567826493" watchObservedRunningTime="2026-10-30 19:45:01.652859526 +0000 UTC m=+780.568466381"
Oct 30 19:45:01.674025 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.673978    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf" podStartSLOduration=1.673965529 podStartE2EDuration="1.673965529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:01.672022425 +0000 UTC m=+780.587629282" watchObservedRunningTime="2026-10-30 19:45:01.673965529 +0000 UTC m=+780.589572384"
Oct 30 19:45:01.695978 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:01.695930    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t" podStartSLOduration=1.695915411 podStartE2EDuration="1.695915411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:01.69484744 +0000 UTC m=+780.610454296" watchObservedRunningTime="2026-10-30 19:45:01.695915411 +0000 UTC m=+780.611522269"
Oct 30 19:45:02.629995 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:02.629960    6030 generic.go:356] "Generic (PLEG): container finished" podID="0ce8f341-0efa-49c8-96aa-27772a49670b" containerID="1471c1a55da29f3faa032be477a7c9b6c123faf98a0535b2c9fc62826dbf6fe0" exitCode=60
Oct 30 19:45:02.630480 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:02.630035    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt" event={"ID":"0ce8f341-0efa-49c8-96aa-27772a49670b","Type":"ContainerDied","Data":"1471c1a55da29f3faa032be477a7c9b6c123faf98a0535b2c9fc62826dbf6fe0"}
Oct 30 19:45:02.630480 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:02.630387    6030 scope.go:122] "RemoveContainer" containerID="1471c1a55da29f3faa032be477a7c9b6c123faf98a0535b2c9fc62826dbf6fe0"
Oct 30 19:45:02.632396 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:02.632357    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t" event={"ID":"d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2","Type":"ContainerDied","Data":"a82d5fed19c3f153335b389f732919283deada290479d73b2488a543f89825ab"}
Oct 30 19:45:02.634572 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:02.632679    6030 generic.go:356] "Generic (PLEG): container finished" podID="d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2" containerID="a82d5fed19c3f153335b389f732919283deada290479d73b2488a543f89825ab" exitCode=60
Oct 30 19:45:02.634572 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:02.632881    6030 scope.go:122] "RemoveContainer" containerID="a82d5fed19c3f153335b389f732919283deada290479d73b2488a543f89825ab"
Oct 30 19:45:02.634905 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:02.634879    6030 generic.go:356] "Generic (PLEG): container finished" podID="d32cf0e1-c0c1-4cbc-98ad-bd177dff4711" containerID="38899f5fbf80244ec3922b5ffcaef288f0ebf98f0863912b257111ec1bb83bb1" exitCode=60
Oct 30 19:45:02.635112 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:02.635090    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf" event={"ID":"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711","Type":"ContainerDied","Data":"38899f5fbf80244ec3922b5ffcaef288f0ebf98f0863912b257111ec1bb83bb1"}
Oct 30 19:45:02.635409 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:02.635389    6030 scope.go:122] "RemoveContainer" containerID="38899f5fbf80244ec3922b5ffcaef288f0ebf98f0863912b257111ec1bb83bb1"
Oct 30 19:45:02.642191 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:02.642166    6030 generic.go:356] "Generic (PLEG): container finished" podID="a9a625e6-791a-4492-9cb5-621e5c52f41a" containerID="0191d5214230e0e1ba54be6ec1f06692faad93b0735363fc6ea6c039fc49b71c" exitCode=0
Oct 30 19:45:02.642310 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:02.642289    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6" event={"ID":"a9a625e6-791a-4492-9cb5-621e5c52f41a","Type":"ContainerDied","Data":"0191d5214230e0e1ba54be6ec1f06692faad93b0735363fc6ea6c039fc49b71c"}
Oct 30 19:45:02.647094 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:02.645765    6030 generic.go:356] "Generic (PLEG): container finished" podID="00a853a9-7fc4-4537-8982-ff5943631c6c" containerID="95b95f8fe50ef4c9273ccbaa1739ef007ea2af67684c982bc369e72eec75fc88" exitCode=60
Oct 30 19:45:02.647094 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:02.645849    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9" event={"ID":"00a853a9-7fc4-4537-8982-ff5943631c6c","Type":"ContainerDied","Data":"95b95f8fe50ef4c9273ccbaa1739ef007ea2af67684c982bc369e72eec75fc88"}
Oct 30 19:45:02.647094 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:02.646028    6030 scope.go:122] "RemoveContainer" containerID="95b95f8fe50ef4c9273ccbaa1739ef007ea2af67684c982bc369e72eec75fc88"
Oct 30 19:45:03.653908 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:03.653872    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt" event={"ID":"0ce8f341-0efa-49c8-96aa-27772a49670b","Type":"ContainerStarted","Data":"bb9f45d26af3125e33c4159a9ccdcce01dfb2bb11b76d02f58f6d9802815bc32"}
Oct 30 19:45:03.655773 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:03.655748    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t" event={"ID":"d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2","Type":"ContainerStarted","Data":"a29a5bf64dda56f2f3cd2bed57c6279f5bf3693f67c973a2bc9601d22138c340"}
Oct 30 19:45:03.657792 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:03.657754    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf" event={"ID":"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711","Type":"ContainerStarted","Data":"53955be0d3ac70369594981770f6ff132519910a6cd3852f34af0df56b1eeb59"}
Oct 30 19:45:03.659906 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:03.659886    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9" event={"ID":"00a853a9-7fc4-4537-8982-ff5943631c6c","Type":"ContainerStarted","Data":"f7a9359b10cc22a6d79d0fb571490435e3106f113e83a884b9946d105d0d6618"}
Oct 30 19:45:03.965626 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:03.965595    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"
Oct 30 19:45:03.993900 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:03.993869    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a625e6-791a-4492-9cb5-621e5c52f41a-kube-api-access-ckvj7\" (UniqueName: \"kubernetes.io/projected/a9a625e6-791a-4492-9cb5-621e5c52f41a-kube-api-access-ckvj7\") pod \"a9a625e6-791a-4492-9cb5-621e5c52f41a\" (UID: \"a9a625e6-791a-4492-9cb5-621e5c52f41a\") "
Oct 30 19:45:03.994119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:03.993986    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a625e6-791a-4492-9cb5-621e5c52f41a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a625e6-791a-4492-9cb5-621e5c52f41a-maas-api-openshift-ca\") pod \"a9a625e6-791a-4492-9cb5-621e5c52f41a\" (UID: \"a9a625e6-791a-4492-9cb5-621e5c52f41a\") "
Oct 30 19:45:03.994439 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:03.994407    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a625e6-791a-4492-9cb5-621e5c52f41a-maas-api-openshift-ca" pod "a9a625e6-791a-4492-9cb5-621e5c52f41a" (UID: "a9a625e6-791a-4492-9cb5-621e5c52f41a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:03.996700 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:03.996677    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a625e6-791a-4492-9cb5-621e5c52f41a-kube-api-access-ckvj7" pod "a9a625e6-791a-4492-9cb5-621e5c52f41a" (UID: "a9a625e6-791a-4492-9cb5-621e5c52f41a"). InnerVolumeSpecName "kube-api-access-ckvj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:04.094999 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.094916    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9a625e6-791a-4492-9cb5-621e5c52f41a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a625e6-791a-4492-9cb5-621e5c52f41a-maas-api-xks-ca\") pod \"a9a625e6-791a-4492-9cb5-621e5c52f41a\" (UID: \"a9a625e6-791a-4492-9cb5-621e5c52f41a\") "
Oct 30 19:45:04.095521 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.095491    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a625e6-791a-4492-9cb5-621e5c52f41a-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:04.095521 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.095514    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckvj7\" (UniqueName: \"kubernetes.io/projected/a9a625e6-791a-4492-9cb5-621e5c52f41a-kube-api-access-ckvj7\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:04.097135 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.097112    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9a625e6-791a-4492-9cb5-621e5c52f41a-maas-api-xks-ca" pod "a9a625e6-791a-4492-9cb5-621e5c52f41a" (UID: "a9a625e6-791a-4492-9cb5-621e5c52f41a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:04.196892 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.196856    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a625e6-791a-4492-9cb5-621e5c52f41a-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:04.290628 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:45:04.290589    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:45:04.725289 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.720599    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6" event={"ID":"a9a625e6-791a-4492-9cb5-621e5c52f41a","Type":"ContainerDied","Data":"3b9e02585080a111e51d4ad70c85019b1f53ee68468ef6ec4203394601e56f9a"}
Oct 30 19:45:04.725289 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.720696    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b9e02585080a111e51d4ad70c85019b1f53ee68468ef6ec4203394601e56f9a"
Oct 30 19:45:04.725289 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.720789    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"
Oct 30 19:45:04.730326 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.730081    6030 generic.go:356] "Generic (PLEG): container finished" podID="00a853a9-7fc4-4537-8982-ff5943631c6c" containerID="f7a9359b10cc22a6d79d0fb571490435e3106f113e83a884b9946d105d0d6618" exitCode=60
Oct 30 19:45:04.730326 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.730258    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9" event={"ID":"00a853a9-7fc4-4537-8982-ff5943631c6c","Type":"ContainerDied","Data":"f7a9359b10cc22a6d79d0fb571490435e3106f113e83a884b9946d105d0d6618"}
Oct 30 19:45:04.730326 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.730323    6030 scope.go:122] "RemoveContainer" containerID="95b95f8fe50ef4c9273ccbaa1739ef007ea2af67684c982bc369e72eec75fc88"
Oct 30 19:45:04.731533 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.730829    6030 scope.go:122] "RemoveContainer" containerID="f7a9359b10cc22a6d79d0fb571490435e3106f113e83a884b9946d105d0d6618"
Oct 30 19:45:04.731533 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:45:04.731289    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9_odh-ai-gateway-infra(00a853a9-7fc4-4537-8982-ff5943631c6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9" podUID="00a853a9-7fc4-4537-8982-ff5943631c6c"
Oct 30 19:45:04.736370 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.736346    6030 generic.go:356] "Generic (PLEG): container finished" podID="0ce8f341-0efa-49c8-96aa-27772a49670b" containerID="bb9f45d26af3125e33c4159a9ccdcce01dfb2bb11b76d02f58f6d9802815bc32" exitCode=60
Oct 30 19:45:04.736480 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.736424    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt" event={"ID":"0ce8f341-0efa-49c8-96aa-27772a49670b","Type":"ContainerDied","Data":"bb9f45d26af3125e33c4159a9ccdcce01dfb2bb11b76d02f58f6d9802815bc32"}
Oct 30 19:45:04.736732 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.736712    6030 scope.go:122] "RemoveContainer" containerID="bb9f45d26af3125e33c4159a9ccdcce01dfb2bb11b76d02f58f6d9802815bc32"
Oct 30 19:45:04.737085 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:45:04.737016    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-9hgvt_odh-ai-gateway-infra(0ce8f341-0efa-49c8-96aa-27772a49670b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt" podUID="0ce8f341-0efa-49c8-96aa-27772a49670b"
Oct 30 19:45:04.738519 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.738498    6030 generic.go:356] "Generic (PLEG): container finished" podID="d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2" containerID="a29a5bf64dda56f2f3cd2bed57c6279f5bf3693f67c973a2bc9601d22138c340" exitCode=60
Oct 30 19:45:04.738603 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.738578    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t" event={"ID":"d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2","Type":"ContainerDied","Data":"a29a5bf64dda56f2f3cd2bed57c6279f5bf3693f67c973a2bc9601d22138c340"}
Oct 30 19:45:04.738813 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.738798    6030 scope.go:122] "RemoveContainer" containerID="a29a5bf64dda56f2f3cd2bed57c6279f5bf3693f67c973a2bc9601d22138c340"
Oct 30 19:45:04.739105 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:45:04.739086    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t_odh-ai-gateway-infra(d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t" podUID="d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2"
Oct 30 19:45:04.740637 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.740615    6030 generic.go:356] "Generic (PLEG): container finished" podID="d32cf0e1-c0c1-4cbc-98ad-bd177dff4711" containerID="53955be0d3ac70369594981770f6ff132519910a6cd3852f34af0df56b1eeb59" exitCode=60
Oct 30 19:45:04.740726 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.740660    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf" event={"ID":"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711","Type":"ContainerDied","Data":"53955be0d3ac70369594981770f6ff132519910a6cd3852f34af0df56b1eeb59"}
Oct 30 19:45:04.740901 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.740887    6030 scope.go:122] "RemoveContainer" containerID="53955be0d3ac70369594981770f6ff132519910a6cd3852f34af0df56b1eeb59"
Oct 30 19:45:04.741128 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:45:04.741110    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf_odh-ai-gateway-infra(d32cf0e1-c0c1-4cbc-98ad-bd177dff4711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf" podUID="d32cf0e1-c0c1-4cbc-98ad-bd177dff4711"
Oct 30 19:45:04.747085 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.747062    6030 scope.go:122] "RemoveContainer" containerID="1471c1a55da29f3faa032be477a7c9b6c123faf98a0535b2c9fc62826dbf6fe0"
Oct 30 19:45:04.760321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.760302    6030 scope.go:122] "RemoveContainer" containerID="a82d5fed19c3f153335b389f732919283deada290479d73b2488a543f89825ab"
Oct 30 19:45:04.772275 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:04.772242    6030 scope.go:122] "RemoveContainer" containerID="38899f5fbf80244ec3922b5ffcaef288f0ebf98f0863912b257111ec1bb83bb1"
Oct 30 19:45:05.749341 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:05.749313    6030 scope.go:122] "RemoveContainer" containerID="f7a9359b10cc22a6d79d0fb571490435e3106f113e83a884b9946d105d0d6618"
Oct 30 19:45:05.749789 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:45:05.749648    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9_odh-ai-gateway-infra(00a853a9-7fc4-4537-8982-ff5943631c6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9" podUID="00a853a9-7fc4-4537-8982-ff5943631c6c"
Oct 30 19:45:05.750974 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:05.750954    6030 scope.go:122] "RemoveContainer" containerID="bb9f45d26af3125e33c4159a9ccdcce01dfb2bb11b76d02f58f6d9802815bc32"
Oct 30 19:45:05.751274 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:45:05.751252    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-9hgvt_odh-ai-gateway-infra(0ce8f341-0efa-49c8-96aa-27772a49670b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt" podUID="0ce8f341-0efa-49c8-96aa-27772a49670b"
Oct 30 19:45:05.752400 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:05.752381    6030 scope.go:122] "RemoveContainer" containerID="a29a5bf64dda56f2f3cd2bed57c6279f5bf3693f67c973a2bc9601d22138c340"
Oct 30 19:45:05.752609 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:45:05.752588    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t_odh-ai-gateway-infra(d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t" podUID="d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2"
Oct 30 19:45:05.753967 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:05.753952    6030 scope.go:122] "RemoveContainer" containerID="53955be0d3ac70369594981770f6ff132519910a6cd3852f34af0df56b1eeb59"
Oct 30 19:45:05.754161 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:45:05.754146    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf_odh-ai-gateway-infra(d32cf0e1-c0c1-4cbc-98ad-bd177dff4711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf" podUID="d32cf0e1-c0c1-4cbc-98ad-bd177dff4711"
Oct 30 19:45:12.296263 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:12.296208    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 19:45:14.236190 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:14.236139    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:45:14.549130 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:14.547410    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:14.574077 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:14.557298    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:14.574077 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:14.572341    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:14.858927 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:14.858829    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:45:14.859230 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:14.859194    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6" podUID="39855966-df67-4ccf-87fe-c5ec63f95cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:15.040069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:15.039560    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:15.041242 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:15.041185    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 19:45:15.159209 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:15.141345    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:15.208144 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:15.185372    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:15.208144 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:15.206680    6030 scope.go:122] "RemoveContainer" containerID="f7a9359b10cc22a6d79d0fb571490435e3106f113e83a884b9946d105d0d6618"
Oct 30 19:45:15.222936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:15.222907    6030 scope.go:122] "RemoveContainer" containerID="53955be0d3ac70369594981770f6ff132519910a6cd3852f34af0df56b1eeb59"
Oct 30 19:45:15.257317 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:15.240241    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:45:15.561873 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:15.461516    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:15.561873 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:15.553172    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:15.568072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:15.564240    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 19:45:15.580314 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:15.580266    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:15.603156 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:15.603115    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:45:15.603475 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:15.603457    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:45:15.603675 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:15.603644    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:45:15.608076 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:15.603849    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:45:15.608308 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:15.608291    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:45:15.696185 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:15.696140    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:16.044652 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.044609    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 19:45:16.077376 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.072472    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:45:16.081022 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.080995    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:45:16.104346 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.104313    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:45:16.177072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.173659    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:45:16.237111 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.194192    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:16.237111 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.214813    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:45:16.237111 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.214874    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:45:16.237111 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.194236    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 19:45:16.295136 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.295018    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:16.300076 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.300017    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:45:16.407073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.376012    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:45:16.429723 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.429341    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9" podUID="409380a1-684a-4be3-9cfc-a9e72c247409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:16.531132 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.524303    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:45:16.531132 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.524373    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq" podUID="487f842d-ad13-45d8-8b55-fbf448ec511d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:16.536249 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.531533    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:16.536249 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.531924    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:45:16.585202 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.557291    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 19:45:16.601085 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.596204    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:45:16.602283 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.602245    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:16.879569 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.879541    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:45:16.928517 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:16.909745    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:45:17.051250 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:17.049486    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:45:17.053276 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:17.053185    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:45:17.062085 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:17.055501    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:45:17.130132 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:17.128922    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:45:17.189068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:17.183963    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf" event={"ID":"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711","Type":"ContainerStarted","Data":"2841135cac01dd056212b8d4d293dd50693378e1829136fb086990cd55455c1c"}
Oct 30 19:45:17.284935 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:17.281832    6030 generic.go:356] "Generic (PLEG): container finished" podID="00a853a9-7fc4-4537-8982-ff5943631c6c" containerID="be3d73d3465fa6179b6a3b64b8dc237f7a8e29db35afff5ff42b1c52ece29cf4" exitCode=60
Oct 30 19:45:17.329062 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:17.329013    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:45:17.335478 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:17.335272    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9" event={"ID":"00a853a9-7fc4-4537-8982-ff5943631c6c","Type":"ContainerDied","Data":"be3d73d3465fa6179b6a3b64b8dc237f7a8e29db35afff5ff42b1c52ece29cf4"}
Oct 30 19:45:17.335986 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:17.335964    6030 scope.go:122] "RemoveContainer" containerID="f7a9359b10cc22a6d79d0fb571490435e3106f113e83a884b9946d105d0d6618"
Oct 30 19:45:17.342382 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:17.342359    6030 scope.go:122] "RemoveContainer" containerID="be3d73d3465fa6179b6a3b64b8dc237f7a8e29db35afff5ff42b1c52ece29cf4"
Oct 30 19:45:17.344842 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:45:17.344808    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9_odh-ai-gateway-infra(00a853a9-7fc4-4537-8982-ff5943631c6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9" podUID="00a853a9-7fc4-4537-8982-ff5943631c6c"
Oct 30 19:45:18.194419 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.194382    6030 scope.go:122] "RemoveContainer" containerID="bb9f45d26af3125e33c4159a9ccdcce01dfb2bb11b76d02f58f6d9802815bc32"
Oct 30 19:45:18.194569 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.194483    6030 scope.go:122] "RemoveContainer" containerID="a29a5bf64dda56f2f3cd2bed57c6279f5bf3693f67c973a2bc9601d22138c340"
Oct 30 19:45:18.292986 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.292959    6030 generic.go:356] "Generic (PLEG): container finished" podID="d32cf0e1-c0c1-4cbc-98ad-bd177dff4711" containerID="2841135cac01dd056212b8d4d293dd50693378e1829136fb086990cd55455c1c" exitCode=60
Oct 30 19:45:18.293123 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.293013    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf" event={"ID":"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711","Type":"ContainerDied","Data":"2841135cac01dd056212b8d4d293dd50693378e1829136fb086990cd55455c1c"}
Oct 30 19:45:18.293123 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.293107    6030 scope.go:122] "RemoveContainer" containerID="53955be0d3ac70369594981770f6ff132519910a6cd3852f34af0df56b1eeb59"
Oct 30 19:45:18.293504 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.293488    6030 scope.go:122] "RemoveContainer" containerID="2841135cac01dd056212b8d4d293dd50693378e1829136fb086990cd55455c1c"
Oct 30 19:45:18.293887 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:45:18.293803    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf_odh-ai-gateway-infra(d32cf0e1-c0c1-4cbc-98ad-bd177dff4711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf" podUID="d32cf0e1-c0c1-4cbc-98ad-bd177dff4711"
Oct 30 19:45:18.330730 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.330697    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"]
Oct 30 19:45:18.331522 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.331499    6030 cpu_manager.go:414] "containerMap: removing container" podUID="a9a625e6-791a-4492-9cb5-621e5c52f41a" containerName="revoke-keys"
Oct 30 19:45:18.331628 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.331523    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a625e6-791a-4492-9cb5-621e5c52f41a" containerName="revoke-keys"
Oct 30 19:45:18.331686 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.331677    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a625e6-791a-4492-9cb5-621e5c52f41a" containerName="revoke-keys"
Oct 30 19:45:18.336742 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.336717    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"
Oct 30 19:45:18.343081 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.342824    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:18.343622 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.343402    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"]
Oct 30 19:45:18.363970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.363944    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf"]
Oct 30 19:45:18.369185 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.369151    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdrqr\" (UniqueName: \"kubernetes.io/projected/7501f0e7-9aea-46de-8455-ab837f1cc08f-kube-api-access-mdrqr\") pod \"maas-api-revoke-sub-3c7c9036-jjwzm\" (UID: \"7501f0e7-9aea-46de-8455-ab837f1cc08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"
Oct 30 19:45:18.369387 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.369360    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7501f0e7-9aea-46de-8455-ab837f1cc08f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c7c9036-jjwzm\" (UID: \"7501f0e7-9aea-46de-8455-ab837f1cc08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"
Oct 30 19:45:18.369469 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.369454    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7501f0e7-9aea-46de-8455-ab837f1cc08f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c7c9036-jjwzm\" (UID: \"7501f0e7-9aea-46de-8455-ab837f1cc08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"
Oct 30 19:45:18.393429 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.393397    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9"]
Oct 30 19:45:18.470375 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.470340    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdrqr\" (UniqueName: \"kubernetes.io/projected/7501f0e7-9aea-46de-8455-ab837f1cc08f-kube-api-access-mdrqr\") pod \"maas-api-revoke-sub-3c7c9036-jjwzm\" (UID: \"7501f0e7-9aea-46de-8455-ab837f1cc08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"
Oct 30 19:45:18.470521 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.470451    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7501f0e7-9aea-46de-8455-ab837f1cc08f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c7c9036-jjwzm\" (UID: \"7501f0e7-9aea-46de-8455-ab837f1cc08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"
Oct 30 19:45:18.470572 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.470523    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7501f0e7-9aea-46de-8455-ab837f1cc08f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c7c9036-jjwzm\" (UID: \"7501f0e7-9aea-46de-8455-ab837f1cc08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"
Oct 30 19:45:18.472241 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.472180    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7501f0e7-9aea-46de-8455-ab837f1cc08f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c7c9036-jjwzm\" (UID: \"7501f0e7-9aea-46de-8455-ab837f1cc08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"
Oct 30 19:45:18.475250 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.475223    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7501f0e7-9aea-46de-8455-ab837f1cc08f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c7c9036-jjwzm\" (UID: \"7501f0e7-9aea-46de-8455-ab837f1cc08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"
Oct 30 19:45:18.490638 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.490606    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdrqr\" (UniqueName: \"kubernetes.io/projected/7501f0e7-9aea-46de-8455-ab837f1cc08f-kube-api-access-mdrqr\") pod \"maas-api-revoke-sub-3c7c9036-jjwzm\" (UID: \"7501f0e7-9aea-46de-8455-ab837f1cc08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"
Oct 30 19:45:18.537279 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.537249    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9"
Oct 30 19:45:18.572380 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.571188    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a853a9-7fc4-4537-8982-ff5943631c6c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a853a9-7fc4-4537-8982-ff5943631c6c-maas-api-service-ca\") pod \"00a853a9-7fc4-4537-8982-ff5943631c6c\" (UID: \"00a853a9-7fc4-4537-8982-ff5943631c6c\") "
Oct 30 19:45:18.572380 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.571253    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a853a9-7fc4-4537-8982-ff5943631c6c-kube-api-access-c7lnr\" (UniqueName: \"kubernetes.io/projected/00a853a9-7fc4-4537-8982-ff5943631c6c-kube-api-access-c7lnr\") pod \"00a853a9-7fc4-4537-8982-ff5943631c6c\" (UID: \"00a853a9-7fc4-4537-8982-ff5943631c6c\") "
Oct 30 19:45:18.573271 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.573216    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a853a9-7fc4-4537-8982-ff5943631c6c-maas-api-service-ca" pod "00a853a9-7fc4-4537-8982-ff5943631c6c" (UID: "00a853a9-7fc4-4537-8982-ff5943631c6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:18.591593 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.591544    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a853a9-7fc4-4537-8982-ff5943631c6c-kube-api-access-c7lnr" pod "00a853a9-7fc4-4537-8982-ff5943631c6c" (UID: "00a853a9-7fc4-4537-8982-ff5943631c6c"). InnerVolumeSpecName "kube-api-access-c7lnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:18.655624 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.655589    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"
Oct 30 19:45:18.673825 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.673796    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a853a9-7fc4-4537-8982-ff5943631c6c-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:18.673825 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.673824    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7lnr\" (UniqueName: \"kubernetes.io/projected/00a853a9-7fc4-4537-8982-ff5943631c6c-kube-api-access-c7lnr\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:18.934624 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:18.934591    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"]
Oct 30 19:45:18.935879 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:45:18.935851    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7501f0e7_9aea_46de_8455_ab837f1cc08f.slice/crio-789627ccbfa0456e593c39698a364c9a504618e228cfabe8bddd646783bb9db4 WatchSource:0}: Error finding container 789627ccbfa0456e593c39698a364c9a504618e228cfabe8bddd646783bb9db4: Status 404 returned error can't find the container with id 789627ccbfa0456e593c39698a364c9a504618e228cfabe8bddd646783bb9db4
Oct 30 19:45:19.300172 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.300134    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm" event={"ID":"7501f0e7-9aea-46de-8455-ab837f1cc08f","Type":"ContainerStarted","Data":"789627ccbfa0456e593c39698a364c9a504618e228cfabe8bddd646783bb9db4"}
Oct 30 19:45:19.301747 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.301723    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9"
Oct 30 19:45:19.301900 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.301761    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9" event={"ID":"00a853a9-7fc4-4537-8982-ff5943631c6c","Type":"ContainerDied","Data":"a59f5addf18330fa2dbc0acd7d22a5d6cef8e46a9e55d64631882d8bd4d7b5ef"}
Oct 30 19:45:19.301900 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.301817    6030 scope.go:122] "RemoveContainer" containerID="be3d73d3465fa6179b6a3b64b8dc237f7a8e29db35afff5ff42b1c52ece29cf4"
Oct 30 19:45:19.304694 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.304668    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt" event={"ID":"0ce8f341-0efa-49c8-96aa-27772a49670b","Type":"ContainerStarted","Data":"6821256152d0dbb39df3a31f4630b91d3754c95d1be605b38043bd7d2211a8d1"}
Oct 30 19:45:19.307006 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.306974    6030 generic.go:356] "Generic (PLEG): container finished" podID="d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2" containerID="3f0eac19b6b7db2a49d796eb0423881e22ab04c4be4ccb29b9e0c0c2acf09223" exitCode=60
Oct 30 19:45:19.307125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.307072    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t" event={"ID":"d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2","Type":"ContainerDied","Data":"3f0eac19b6b7db2a49d796eb0423881e22ab04c4be4ccb29b9e0c0c2acf09223"}
Oct 30 19:45:19.307747 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.307440    6030 scope.go:122] "RemoveContainer" containerID="3f0eac19b6b7db2a49d796eb0423881e22ab04c4be4ccb29b9e0c0c2acf09223"
Oct 30 19:45:19.307747 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:45:19.307707    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t_odh-ai-gateway-infra(d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t" podUID="d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2"
Oct 30 19:45:19.314358 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.314336    6030 scope.go:122] "RemoveContainer" containerID="a29a5bf64dda56f2f3cd2bed57c6279f5bf3693f67c973a2bc9601d22138c340"
Oct 30 19:45:19.343503 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.343466    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9"]
Oct 30 19:45:19.347258 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.347219    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889825-rddd9"]
Oct 30 19:45:19.480980 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.480898    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf"
Oct 30 19:45:19.583800 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.583772    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32cf0e1-c0c1-4cbc-98ad-bd177dff4711-kube-api-access-2895p\" (UniqueName: \"kubernetes.io/projected/d32cf0e1-c0c1-4cbc-98ad-bd177dff4711-kube-api-access-2895p\") pod \"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711\" (UID: \"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711\") "
Oct 30 19:45:19.583995 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.583864    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d32cf0e1-c0c1-4cbc-98ad-bd177dff4711-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32cf0e1-c0c1-4cbc-98ad-bd177dff4711-maas-api-service-ca\") pod \"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711\" (UID: \"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711\") "
Oct 30 19:45:19.584284 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.584260    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32cf0e1-c0c1-4cbc-98ad-bd177dff4711-maas-api-service-ca" pod "d32cf0e1-c0c1-4cbc-98ad-bd177dff4711" (UID: "d32cf0e1-c0c1-4cbc-98ad-bd177dff4711"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:19.584376 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.584364    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32cf0e1-c0c1-4cbc-98ad-bd177dff4711-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.586191 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.586170    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32cf0e1-c0c1-4cbc-98ad-bd177dff4711-kube-api-access-2895p" pod "d32cf0e1-c0c1-4cbc-98ad-bd177dff4711" (UID: "d32cf0e1-c0c1-4cbc-98ad-bd177dff4711"). InnerVolumeSpecName "kube-api-access-2895p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:19.719713 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:19.719675    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2895p\" (UniqueName: \"kubernetes.io/projected/d32cf0e1-c0c1-4cbc-98ad-bd177dff4711-kube-api-access-2895p\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:20.317406 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.317309    6030 generic.go:356] "Generic (PLEG): container finished" podID="7501f0e7-9aea-46de-8455-ab837f1cc08f" containerID="e898e0c10951c4ec750e444bef2427a87d3f2f7ea06ed80d1ddd05caa0f9859e" exitCode=0
Oct 30 19:45:20.317406 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.317350    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm" event={"ID":"7501f0e7-9aea-46de-8455-ab837f1cc08f","Type":"ContainerDied","Data":"e898e0c10951c4ec750e444bef2427a87d3f2f7ea06ed80d1ddd05caa0f9859e"}
Oct 30 19:45:20.320267 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.320242    6030 generic.go:356] "Generic (PLEG): container finished" podID="0ce8f341-0efa-49c8-96aa-27772a49670b" containerID="6821256152d0dbb39df3a31f4630b91d3754c95d1be605b38043bd7d2211a8d1" exitCode=60
Oct 30 19:45:20.320420 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.320301    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt" event={"ID":"0ce8f341-0efa-49c8-96aa-27772a49670b","Type":"ContainerDied","Data":"6821256152d0dbb39df3a31f4630b91d3754c95d1be605b38043bd7d2211a8d1"}
Oct 30 19:45:20.320420 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.320345    6030 scope.go:122] "RemoveContainer" containerID="bb9f45d26af3125e33c4159a9ccdcce01dfb2bb11b76d02f58f6d9802815bc32"
Oct 30 19:45:20.320710 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.320693    6030 scope.go:122] "RemoveContainer" containerID="6821256152d0dbb39df3a31f4630b91d3754c95d1be605b38043bd7d2211a8d1"
Oct 30 19:45:20.321007 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:45:20.320985    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-9hgvt_odh-ai-gateway-infra(0ce8f341-0efa-49c8-96aa-27772a49670b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt" podUID="0ce8f341-0efa-49c8-96aa-27772a49670b"
Oct 30 19:45:20.323548 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.323529    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf" event={"ID":"d32cf0e1-c0c1-4cbc-98ad-bd177dff4711","Type":"ContainerDied","Data":"c86447df1190d3ace58ce920632d351569c561f8d2cf90a00d649f88c9daa351"}
Oct 30 19:45:20.323661 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.323578    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf"
Oct 30 19:45:20.333410 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.333388    6030 scope.go:122] "RemoveContainer" containerID="2841135cac01dd056212b8d4d293dd50693378e1829136fb086990cd55455c1c"
Oct 30 19:45:20.338835 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.338036    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt"]
Oct 30 19:45:20.359346 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.359313    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf"]
Oct 30 19:45:20.368189 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.368163    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889825-m8wqf"]
Oct 30 19:45:20.399670 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.399639    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t"]
Oct 30 19:45:20.558678 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.558652    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t"
Oct 30 19:45:20.633012 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.632584    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2-maas-api-service-ca\") pod \"d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2\" (UID: \"d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2\") "
Oct 30 19:45:20.633012 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.632863    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2-kube-api-access-5pdtc\" (UniqueName: \"kubernetes.io/projected/d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2-kube-api-access-5pdtc\") pod \"d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2\" (UID: \"d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2\") "
Oct 30 19:45:20.633898 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.633866    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2-maas-api-service-ca" pod "d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2" (UID: "d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:20.637191 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.636997    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2-kube-api-access-5pdtc" pod "d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2" (UID: "d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2"). InnerVolumeSpecName "kube-api-access-5pdtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:20.734383 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.734353    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pdtc\" (UniqueName: \"kubernetes.io/projected/d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2-kube-api-access-5pdtc\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:20.734383 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:20.734381    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:21.203138 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.203106    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a853a9-7fc4-4537-8982-ff5943631c6c" path="/var/lib/kubelet/pods/00a853a9-7fc4-4537-8982-ff5943631c6c/volumes"
Oct 30 19:45:21.203578 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.203563    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d32cf0e1-c0c1-4cbc-98ad-bd177dff4711" path="/var/lib/kubelet/pods/d32cf0e1-c0c1-4cbc-98ad-bd177dff4711/volumes"
Oct 30 19:45:21.341404 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.341370    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t"
Oct 30 19:45:21.341692 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.341661    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t" event={"ID":"d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2","Type":"ContainerDied","Data":"17444ab4b5a0ebf868184848d68ba572abc52658ddd9741e7407442b8746f44e"}
Oct 30 19:45:21.341927 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.341906    6030 scope.go:122] "RemoveContainer" containerID="3f0eac19b6b7db2a49d796eb0423881e22ab04c4be4ccb29b9e0c0c2acf09223"
Oct 30 19:45:21.380523 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.380450    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t"]
Oct 30 19:45:21.385362 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.385327    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889825-dqk5t"]
Oct 30 19:45:21.511089 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.511036    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt"
Oct 30 19:45:21.542847 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.542824    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"
Oct 30 19:45:21.543859 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.543835    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ce8f341-0efa-49c8-96aa-27772a49670b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce8f341-0efa-49c8-96aa-27772a49670b-maas-api-service-ca\") pod \"0ce8f341-0efa-49c8-96aa-27772a49670b\" (UID: \"0ce8f341-0efa-49c8-96aa-27772a49670b\") "
Oct 30 19:45:21.543997 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.543958    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce8f341-0efa-49c8-96aa-27772a49670b-kube-api-access-nj4zs\" (UniqueName: \"kubernetes.io/projected/0ce8f341-0efa-49c8-96aa-27772a49670b-kube-api-access-nj4zs\") pod \"0ce8f341-0efa-49c8-96aa-27772a49670b\" (UID: \"0ce8f341-0efa-49c8-96aa-27772a49670b\") "
Oct 30 19:45:21.544359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.544335    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce8f341-0efa-49c8-96aa-27772a49670b-maas-api-service-ca" pod "0ce8f341-0efa-49c8-96aa-27772a49670b" (UID: "0ce8f341-0efa-49c8-96aa-27772a49670b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:21.544474 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.544458    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce8f341-0efa-49c8-96aa-27772a49670b-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:21.547780 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.547753    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce8f341-0efa-49c8-96aa-27772a49670b-kube-api-access-nj4zs" pod "0ce8f341-0efa-49c8-96aa-27772a49670b" (UID: "0ce8f341-0efa-49c8-96aa-27772a49670b"). InnerVolumeSpecName "kube-api-access-nj4zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:21.645071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.645015    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7501f0e7-9aea-46de-8455-ab837f1cc08f-kube-api-access-mdrqr\" (UniqueName: \"kubernetes.io/projected/7501f0e7-9aea-46de-8455-ab837f1cc08f-kube-api-access-mdrqr\") pod \"7501f0e7-9aea-46de-8455-ab837f1cc08f\" (UID: \"7501f0e7-9aea-46de-8455-ab837f1cc08f\") "
Oct 30 19:45:21.645224 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.645121    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7501f0e7-9aea-46de-8455-ab837f1cc08f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7501f0e7-9aea-46de-8455-ab837f1cc08f-maas-api-openshift-ca\") pod \"7501f0e7-9aea-46de-8455-ab837f1cc08f\" (UID: \"7501f0e7-9aea-46de-8455-ab837f1cc08f\") "
Oct 30 19:45:21.645224 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.645152    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7501f0e7-9aea-46de-8455-ab837f1cc08f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7501f0e7-9aea-46de-8455-ab837f1cc08f-maas-api-xks-ca\") pod \"7501f0e7-9aea-46de-8455-ab837f1cc08f\" (UID: \"7501f0e7-9aea-46de-8455-ab837f1cc08f\") "
Oct 30 19:45:21.645551 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.645521    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7501f0e7-9aea-46de-8455-ab837f1cc08f-maas-api-openshift-ca" pod "7501f0e7-9aea-46de-8455-ab837f1cc08f" (UID: "7501f0e7-9aea-46de-8455-ab837f1cc08f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:21.645674 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.645618    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj4zs\" (UniqueName: \"kubernetes.io/projected/0ce8f341-0efa-49c8-96aa-27772a49670b-kube-api-access-nj4zs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:21.645674 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.645635    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7501f0e7-9aea-46de-8455-ab837f1cc08f-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:21.647382 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.647360    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7501f0e7-9aea-46de-8455-ab837f1cc08f-maas-api-xks-ca" pod "7501f0e7-9aea-46de-8455-ab837f1cc08f" (UID: "7501f0e7-9aea-46de-8455-ab837f1cc08f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:21.647454 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.647397    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7501f0e7-9aea-46de-8455-ab837f1cc08f-kube-api-access-mdrqr" pod "7501f0e7-9aea-46de-8455-ab837f1cc08f" (UID: "7501f0e7-9aea-46de-8455-ab837f1cc08f"). InnerVolumeSpecName "kube-api-access-mdrqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:21.746943 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.746865    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7501f0e7-9aea-46de-8455-ab837f1cc08f-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:21.746943 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:21.746892    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdrqr\" (UniqueName: \"kubernetes.io/projected/7501f0e7-9aea-46de-8455-ab837f1cc08f-kube-api-access-mdrqr\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:22.353549 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:22.353519    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"
Oct 30 19:45:22.353741 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:22.353512    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm" event={"ID":"7501f0e7-9aea-46de-8455-ab837f1cc08f","Type":"ContainerDied","Data":"789627ccbfa0456e593c39698a364c9a504618e228cfabe8bddd646783bb9db4"}
Oct 30 19:45:22.353741 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:22.353645    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="789627ccbfa0456e593c39698a364c9a504618e228cfabe8bddd646783bb9db4"
Oct 30 19:45:22.354872 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:22.354849    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt"
Oct 30 19:45:22.355003 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:22.354851    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt" event={"ID":"0ce8f341-0efa-49c8-96aa-27772a49670b","Type":"ContainerDied","Data":"100f0989f09f20aaa3435d2806f2575f5222edc8215b53cdcadfb322479e1de7"}
Oct 30 19:45:22.355003 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:22.354983    6030 scope.go:122] "RemoveContainer" containerID="6821256152d0dbb39df3a31f4630b91d3754c95d1be605b38043bd7d2211a8d1"
Oct 30 19:45:22.387229 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:22.387197    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt"]
Oct 30 19:45:22.392441 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:22.392413    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-9hgvt"]
Oct 30 19:45:23.200951 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:23.200920    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ce8f341-0efa-49c8-96aa-27772a49670b" path="/var/lib/kubelet/pods/0ce8f341-0efa-49c8-96aa-27772a49670b/volumes"
Oct 30 19:45:23.201446 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:23.201429    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2" path="/var/lib/kubelet/pods/d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2/volumes"
Oct 30 19:45:24.946691 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.946652    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"]
Oct 30 19:45:24.947215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947196    6030 cpu_manager.go:414] "containerMap: removing container" podUID="d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2" containerName="cleanup"
Oct 30 19:45:24.947290 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947219    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2" containerName="cleanup"
Oct 30 19:45:24.947290 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947235    6030 cpu_manager.go:414] "containerMap: removing container" podUID="00a853a9-7fc4-4537-8982-ff5943631c6c" containerName="cleanup"
Oct 30 19:45:24.947290 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947240    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a853a9-7fc4-4537-8982-ff5943631c6c" containerName="cleanup"
Oct 30 19:45:24.947290 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947270    6030 cpu_manager.go:414] "containerMap: removing container" podUID="0ce8f341-0efa-49c8-96aa-27772a49670b" containerName="cleanup"
Oct 30 19:45:24.947290 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947278    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce8f341-0efa-49c8-96aa-27772a49670b" containerName="cleanup"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947292    6030 cpu_manager.go:414] "containerMap: removing container" podUID="d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2" containerName="cleanup"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947300    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2" containerName="cleanup"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947308    6030 cpu_manager.go:414] "containerMap: removing container" podUID="00a853a9-7fc4-4537-8982-ff5943631c6c" containerName="cleanup"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947312    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a853a9-7fc4-4537-8982-ff5943631c6c" containerName="cleanup"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947322    6030 cpu_manager.go:414] "containerMap: removing container" podUID="d32cf0e1-c0c1-4cbc-98ad-bd177dff4711" containerName="cleanup"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947326    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32cf0e1-c0c1-4cbc-98ad-bd177dff4711" containerName="cleanup"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947333    6030 cpu_manager.go:414] "containerMap: removing container" podUID="0ce8f341-0efa-49c8-96aa-27772a49670b" containerName="cleanup"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947337    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce8f341-0efa-49c8-96aa-27772a49670b" containerName="cleanup"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947341    6030 cpu_manager.go:414] "containerMap: removing container" podUID="00a853a9-7fc4-4537-8982-ff5943631c6c" containerName="cleanup"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947347    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a853a9-7fc4-4537-8982-ff5943631c6c" containerName="cleanup"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947359    6030 cpu_manager.go:414] "containerMap: removing container" podUID="d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2" containerName="cleanup"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947363    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2" containerName="cleanup"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947379    6030 cpu_manager.go:414] "containerMap: removing container" podUID="d32cf0e1-c0c1-4cbc-98ad-bd177dff4711" containerName="cleanup"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947383    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32cf0e1-c0c1-4cbc-98ad-bd177dff4711" containerName="cleanup"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947388    6030 cpu_manager.go:414] "containerMap: removing container" podUID="7501f0e7-9aea-46de-8455-ab837f1cc08f" containerName="revoke-keys"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947392    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7501f0e7-9aea-46de-8455-ab837f1cc08f" containerName="revoke-keys"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947476    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a853a9-7fc4-4537-8982-ff5943631c6c" containerName="cleanup"
Oct 30 19:45:24.947484 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947489    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce8f341-0efa-49c8-96aa-27772a49670b" containerName="cleanup"
Oct 30 19:45:24.948284 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947498    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2" containerName="cleanup"
Oct 30 19:45:24.948284 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947506    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a853a9-7fc4-4537-8982-ff5943631c6c" containerName="cleanup"
Oct 30 19:45:24.948284 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947519    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32cf0e1-c0c1-4cbc-98ad-bd177dff4711" containerName="cleanup"
Oct 30 19:45:24.948284 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947527    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce8f341-0efa-49c8-96aa-27772a49670b" containerName="cleanup"
Oct 30 19:45:24.948284 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947536    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a853a9-7fc4-4537-8982-ff5943631c6c" containerName="cleanup"
Oct 30 19:45:24.948284 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947544    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2" containerName="cleanup"
Oct 30 19:45:24.948284 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947558    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32cf0e1-c0c1-4cbc-98ad-bd177dff4711" containerName="cleanup"
Oct 30 19:45:24.948284 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.947566    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="7501f0e7-9aea-46de-8455-ab837f1cc08f" containerName="revoke-keys"
Oct 30 19:45:24.952829 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.952805    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"
Oct 30 19:45:24.958148 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.958121    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 19:45:24.958280 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.958192    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:24.958347 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.958285    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:45:24.958910 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.958888    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:45:24.959958 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.959934    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"]
Oct 30 19:45:24.988258 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.988226    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef436d3c-4p4wl\" (UID: \"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"
Oct 30 19:45:24.988421 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.988318    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k8b7\" (UniqueName: \"kubernetes.io/projected/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-kube-api-access-5k8b7\") pod \"maas-api-revoke-sub-ef436d3c-4p4wl\" (UID: \"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"
Oct 30 19:45:24.988421 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:24.988353    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef436d3c-4p4wl\" (UID: \"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"
Oct 30 19:45:25.089447 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:25.089410    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8b7\" (UniqueName: \"kubernetes.io/projected/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-kube-api-access-5k8b7\") pod \"maas-api-revoke-sub-ef436d3c-4p4wl\" (UID: \"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"
Oct 30 19:45:25.089619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:25.089454    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef436d3c-4p4wl\" (UID: \"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"
Oct 30 19:45:25.089619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:25.089595    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef436d3c-4p4wl\" (UID: \"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"
Oct 30 19:45:25.090156 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:25.090133    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef436d3c-4p4wl\" (UID: \"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"
Oct 30 19:45:25.092098 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:25.092074    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef436d3c-4p4wl\" (UID: \"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"
Oct 30 19:45:25.103308 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:25.103288    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k8b7\" (UniqueName: \"kubernetes.io/projected/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-kube-api-access-5k8b7\") pod \"maas-api-revoke-sub-ef436d3c-4p4wl\" (UID: \"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"
Oct 30 19:45:25.272491 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:25.272400    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"
Oct 30 19:45:25.575113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:25.575081    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"]
Oct 30 19:45:25.576440 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:45:25.576412    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e221623_3b0d_4d13_b2e5_922a6ab8f9b4.slice/crio-515fbea175dbcd6212bf163581c410945f527983d98c253f18efe28e560f333b WatchSource:0}: Error finding container 515fbea175dbcd6212bf163581c410945f527983d98c253f18efe28e560f333b: Status 404 returned error can't find the container with id 515fbea175dbcd6212bf163581c410945f527983d98c253f18efe28e560f333b
Oct 30 19:45:26.388346 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:26.388222    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl" event={"ID":"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4","Type":"ContainerStarted","Data":"c1f56dd77a9c0c832ef4d035ad4aae87a81131544d332c46d5401c983b6283b4"}
Oct 30 19:45:26.388346 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:26.388282    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl" event={"ID":"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4","Type":"ContainerStarted","Data":"515fbea175dbcd6212bf163581c410945f527983d98c253f18efe28e560f333b"}
Oct 30 19:45:26.410093 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:26.410005    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl" podStartSLOduration=2.409991142 podStartE2EDuration="2.409991142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:26.4087442 +0000 UTC m=+805.324351056" watchObservedRunningTime="2026-10-30 19:45:26.409991142 +0000 UTC m=+805.325597998"
Oct 30 19:45:27.251080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.251010    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"]
Oct 30 19:45:27.251605 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.251585    6030 cpu_manager.go:414] "containerMap: removing container" podUID="0ce8f341-0efa-49c8-96aa-27772a49670b" containerName="cleanup"
Oct 30 19:45:27.251605 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.251606    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce8f341-0efa-49c8-96aa-27772a49670b" containerName="cleanup"
Oct 30 19:45:27.251757 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.251637    6030 cpu_manager.go:414] "containerMap: removing container" podUID="d32cf0e1-c0c1-4cbc-98ad-bd177dff4711" containerName="cleanup"
Oct 30 19:45:27.251757 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.251645    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32cf0e1-c0c1-4cbc-98ad-bd177dff4711" containerName="cleanup"
Oct 30 19:45:27.251757 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.251737    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce8f341-0efa-49c8-96aa-27772a49670b" containerName="cleanup"
Oct 30 19:45:27.251757 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.251748    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ed452d-ce5e-4aa8-93f0-e7204bf22fd2" containerName="cleanup"
Oct 30 19:45:27.252008 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.251762    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32cf0e1-c0c1-4cbc-98ad-bd177dff4711" containerName="cleanup"
Oct 30 19:45:27.255589 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.255563    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.264266 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.264242    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7a745c\"/\"distinct-w1-7a745c-kserve-self-signed-certs\""
Oct 30 19:45:27.272251 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.270756    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"]
Oct 30 19:45:27.354222 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.354175    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-model-cache\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.354425 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.354249    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-home\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.354425 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.354278    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db52234-3bbf-4a48-b905-dec230b1b3f8-tls-certs\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.354425 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.354414    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-tmp-dir\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.354559 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.354481    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-dshm\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.354737 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.354708    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48l9f\" (UniqueName: \"kubernetes.io/projected/2db52234-3bbf-4a48-b905-dec230b1b3f8-kube-api-access-48l9f\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.448111 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.445469    6030 generic.go:356] "Generic (PLEG): container finished" podID="5e221623-3b0d-4d13-b2e5-922a6ab8f9b4" containerID="c1f56dd77a9c0c832ef4d035ad4aae87a81131544d332c46d5401c983b6283b4" exitCode=0
Oct 30 19:45:27.448111 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.445560    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl" event={"ID":"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4","Type":"ContainerDied","Data":"c1f56dd77a9c0c832ef4d035ad4aae87a81131544d332c46d5401c983b6283b4"}
Oct 30 19:45:27.465069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.462938    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-tmp-dir\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.465069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.463024    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-dshm\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.465069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.463195    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48l9f\" (UniqueName: \"kubernetes.io/projected/2db52234-3bbf-4a48-b905-dec230b1b3f8-kube-api-access-48l9f\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.465069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.463256    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-model-cache\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.465069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.463302    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-home\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.465069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.463330    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db52234-3bbf-4a48-b905-dec230b1b3f8-tls-certs\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.465069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.464463    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-tmp-dir\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.465069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.464970    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-model-cache\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.465627 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.465222    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-home\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.485676 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.485645    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db52234-3bbf-4a48-b905-dec230b1b3f8-tls-certs\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.494703 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.494673    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-dshm\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.529920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.529837    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48l9f\" (UniqueName: \"kubernetes.io/projected/2db52234-3bbf-4a48-b905-dec230b1b3f8-kube-api-access-48l9f\") pod \"distinct-w1-7a745c-kserve-55676d64c-58bw5\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.569606 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.569579    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:27.869898 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:27.869818    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"]
Oct 30 19:45:27.873011 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:45:27.872977    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2db52234_3bbf_4a48_b905_dec230b1b3f8.slice/crio-2c4e26d2273ed905ff14379c2e701e41a0dcf4f86cc29acd57d2e08a1a5c0a4f WatchSource:0}: Error finding container 2c4e26d2273ed905ff14379c2e701e41a0dcf4f86cc29acd57d2e08a1a5c0a4f: Status 404 returned error can't find the container with id 2c4e26d2273ed905ff14379c2e701e41a0dcf4f86cc29acd57d2e08a1a5c0a4f
Oct 30 19:45:28.452921 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:28.452876    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5" event={"ID":"2db52234-3bbf-4a48-b905-dec230b1b3f8","Type":"ContainerStarted","Data":"7843e1871620b86d08401fb524e26f52221f23c658cf0fd2dde10f459ae12008"}
Oct 30 19:45:28.452921 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:28.452916    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5" event={"ID":"2db52234-3bbf-4a48-b905-dec230b1b3f8","Type":"ContainerStarted","Data":"2c4e26d2273ed905ff14379c2e701e41a0dcf4f86cc29acd57d2e08a1a5c0a4f"}
Oct 30 19:45:28.484501 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:28.484436    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5" podStartSLOduration=1.484415465 podStartE2EDuration="1.484415465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:28.479968693 +0000 UTC m=+807.395575551" watchObservedRunningTime="2026-10-30 19:45:28.484415465 +0000 UTC m=+807.400022322"
Oct 30 19:45:28.571241 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:28.571212    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:28.591307 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:28.591285    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"
Oct 30 19:45:28.679974 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:28.679946    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-maas-api-openshift-ca\") pod \"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4\" (UID: \"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4\") "
Oct 30 19:45:28.680190 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:28.680062    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-kube-api-access-5k8b7\" (UniqueName: \"kubernetes.io/projected/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-kube-api-access-5k8b7\") pod \"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4\" (UID: \"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4\") "
Oct 30 19:45:28.680190 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:28.680100    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-maas-api-xks-ca\") pod \"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4\" (UID: \"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4\") "
Oct 30 19:45:28.681019 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:28.680995    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-maas-api-openshift-ca" pod "5e221623-3b0d-4d13-b2e5-922a6ab8f9b4" (UID: "5e221623-3b0d-4d13-b2e5-922a6ab8f9b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:28.686512 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:28.686486    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-kube-api-access-5k8b7" pod "5e221623-3b0d-4d13-b2e5-922a6ab8f9b4" (UID: "5e221623-3b0d-4d13-b2e5-922a6ab8f9b4"). InnerVolumeSpecName "kube-api-access-5k8b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:28.693202 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:28.693176    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-maas-api-xks-ca" pod "5e221623-3b0d-4d13-b2e5-922a6ab8f9b4" (UID: "5e221623-3b0d-4d13-b2e5-922a6ab8f9b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:28.781904 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:28.781824    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k8b7\" (UniqueName: \"kubernetes.io/projected/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-kube-api-access-5k8b7\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:28.781904 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:28.781864    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:28.781904 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:28.781880    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:29.496163 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:29.496080    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"
Oct 30 19:45:29.498436 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:29.498387    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl" event={"ID":"5e221623-3b0d-4d13-b2e5-922a6ab8f9b4","Type":"ContainerDied","Data":"515fbea175dbcd6212bf163581c410945f527983d98c253f18efe28e560f333b"}
Oct 30 19:45:29.509578 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:29.505989    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="515fbea175dbcd6212bf163581c410945f527983d98c253f18efe28e560f333b"
Oct 30 19:45:33.161072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:33.160470    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 19:45:33.576135 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:33.573154    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:33.934072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:33.933157    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:34.069075 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:34.068327    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:34.073071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:34.069456    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:34.201142 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:34.165585    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:34.315076 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:34.308150    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:34.606123 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:34.603782    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:35.040073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:34.940160    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:35.040073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:34.940991    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:45:35.040333 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:35.040165    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:35.084191 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:35.084143    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:35.102067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:35.091311    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 19:45:35.115158 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:35.102663    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:45:35.130091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:35.129146    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:45:35.192071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:35.191261    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:35.225070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:35.204546    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:35.297128 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:35.293135    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:45:35.322070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:35.309151    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 19:45:35.485073 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:45:35.480266    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:45:35.535084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:35.497272    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:45:35.535084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:35.497339    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:45:35.622816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:35.616531    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:35.622816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:35.616878    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:45:35.622816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:35.617544    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:45:35.712072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:35.711568    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:36.006084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:35.989792    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:45:36.006084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:35.991147    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:36.044074 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.043136    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:36.044074 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.043598    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:45:36.098070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.097400    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 19:45:36.109604 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.108349    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 19:45:36.244126 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.228971    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:36.244126 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.232704    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 19:45:36.264117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.264087    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:45:36.396073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.380549    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:45:36.549247 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.489153    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:36.549247 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.489189    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 19:45:36.576167 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.550400    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"]
Oct 30 19:45:36.576167 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.551186    6030 cpu_manager.go:414] "containerMap: removing container" podUID="5e221623-3b0d-4d13-b2e5-922a6ab8f9b4" containerName="revoke-keys"
Oct 30 19:45:36.576167 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.551205    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e221623-3b0d-4d13-b2e5-922a6ab8f9b4" containerName="revoke-keys"
Oct 30 19:45:36.576167 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.551336    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e221623-3b0d-4d13-b2e5-922a6ab8f9b4" containerName="revoke-keys"
Oct 30 19:45:36.579060 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.579010    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"
Oct 30 19:45:36.623312 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.623280    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:45:36.623631 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.623616    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 19:45:36.623797 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.623784    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:36.623925 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.623915    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:45:36.624915 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.624890    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"]
Oct 30 19:45:36.625213 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.625187    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:36.626622 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.626595    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 19:45:36.627384 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.627363    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:45:36.715221 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.715180    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-346ed860-ntgg9\" (UID: \"ca219c8b-7690-45c6-9ed9-dd04a5fd458c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"
Oct 30 19:45:36.720086 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.718164    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bnz\" (UniqueName: \"kubernetes.io/projected/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-kube-api-access-v9bnz\") pod \"maas-api-revoke-sub-346ed860-ntgg9\" (UID: \"ca219c8b-7690-45c6-9ed9-dd04a5fd458c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"
Oct 30 19:45:36.785202 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.767322    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-346ed860-ntgg9\" (UID: \"ca219c8b-7690-45c6-9ed9-dd04a5fd458c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"
Oct 30 19:45:36.898907 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.898866    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-346ed860-ntgg9\" (UID: \"ca219c8b-7690-45c6-9ed9-dd04a5fd458c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"
Oct 30 19:45:36.908287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.908255    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bnz\" (UniqueName: \"kubernetes.io/projected/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-kube-api-access-v9bnz\") pod \"maas-api-revoke-sub-346ed860-ntgg9\" (UID: \"ca219c8b-7690-45c6-9ed9-dd04a5fd458c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"
Oct 30 19:45:36.908661 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.908623    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-346ed860-ntgg9\" (UID: \"ca219c8b-7690-45c6-9ed9-dd04a5fd458c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"
Oct 30 19:45:36.928786 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.928742    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-346ed860-ntgg9\" (UID: \"ca219c8b-7690-45c6-9ed9-dd04a5fd458c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"
Oct 30 19:45:37.163143 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:36.969030    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-346ed860-ntgg9\" (UID: \"ca219c8b-7690-45c6-9ed9-dd04a5fd458c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"
Oct 30 19:45:37.184070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:37.066116    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:37.221102 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:37.066157    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:37.283069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:37.158208    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:45:37.295069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:37.292544    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 19:45:37.295069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:37.249806    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:45:37.295069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:37.293107    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh" podUID="becea00a-6860-4f4e-830f-6f504d0cf8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:37.295069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:37.273202    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 19:45:37.295443 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:37.186141    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bnz\" (UniqueName: \"kubernetes.io/projected/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-kube-api-access-v9bnz\") pod \"maas-api-revoke-sub-346ed860-ntgg9\" (UID: \"ca219c8b-7690-45c6-9ed9-dd04a5fd458c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"
Oct 30 19:45:37.353070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:37.320383    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:37.430122 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:37.429585    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:45:37.435363 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:37.435329    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"
Oct 30 19:45:37.542213 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:37.542174    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:37.615149 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:37.608080    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:37.631362 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:37.631319    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:45:37.632806 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:37.632774    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:37.698623 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:37.698584    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:37.721082 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:37.716344    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:45:38.114009 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:38.112621    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:45:38.153858 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:38.139342    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:45:38.225402 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:38.221604    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:38.258588 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:38.258559    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:45:38.275299 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:38.275237    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:38.336597 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:38.334674    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:45:38.339007 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:38.338980    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:45:38.341231 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:38.341199    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 19:45:38.350314 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:38.348641    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:45:38.372358 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:38.372330    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:45:38.383286 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:38.383244    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:45:38.388681 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:38.388651    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:45:38.392348 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:38.392325    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"]
Oct 30 19:45:39.339226 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:39.339193    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9" event={"ID":"ca219c8b-7690-45c6-9ed9-dd04a5fd458c","Type":"ContainerStarted","Data":"c8f141f95ecb88d14620e5b54b3c21ccff310961ab2c54436e1e9074b72cb957"}
Oct 30 19:45:39.339681 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:39.339255    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9" event={"ID":"ca219c8b-7690-45c6-9ed9-dd04a5fd458c","Type":"ContainerStarted","Data":"4b970f474b5760709ae965d8293340f5fdfbea2e61aa851bd709b6e1d8c10939"}
Oct 30 19:45:39.369217 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:39.369164    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9" podStartSLOduration=3.3691487159999998 podStartE2EDuration="3.369148716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:39.365012592 +0000 UTC m=+818.280619449" watchObservedRunningTime="2026-10-30 19:45:39.369148716 +0000 UTC m=+818.284755572"
Oct 30 19:45:40.396466 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:40.396408    6030 generic.go:356] "Generic (PLEG): container finished" podID="ca219c8b-7690-45c6-9ed9-dd04a5fd458c" containerID="c8f141f95ecb88d14620e5b54b3c21ccff310961ab2c54436e1e9074b72cb957" exitCode=0
Oct 30 19:45:40.396897 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:40.396558    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9" event={"ID":"ca219c8b-7690-45c6-9ed9-dd04a5fd458c","Type":"ContainerDied","Data":"c8f141f95ecb88d14620e5b54b3c21ccff310961ab2c54436e1e9074b72cb957"}
Oct 30 19:45:41.576242 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:41.576212    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"
Oct 30 19:45:41.657926 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:41.657837    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-maas-api-xks-ca\") pod \"ca219c8b-7690-45c6-9ed9-dd04a5fd458c\" (UID: \"ca219c8b-7690-45c6-9ed9-dd04a5fd458c\") "
Oct 30 19:45:41.657926 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:41.657892    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-maas-api-openshift-ca\") pod \"ca219c8b-7690-45c6-9ed9-dd04a5fd458c\" (UID: \"ca219c8b-7690-45c6-9ed9-dd04a5fd458c\") "
Oct 30 19:45:41.658179 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:41.658054    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-kube-api-access-v9bnz\" (UniqueName: \"kubernetes.io/projected/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-kube-api-access-v9bnz\") pod \"ca219c8b-7690-45c6-9ed9-dd04a5fd458c\" (UID: \"ca219c8b-7690-45c6-9ed9-dd04a5fd458c\") "
Oct 30 19:45:41.658356 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:41.658326    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-maas-api-openshift-ca" pod "ca219c8b-7690-45c6-9ed9-dd04a5fd458c" (UID: "ca219c8b-7690-45c6-9ed9-dd04a5fd458c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:41.660426 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:41.660401    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-maas-api-xks-ca" pod "ca219c8b-7690-45c6-9ed9-dd04a5fd458c" (UID: "ca219c8b-7690-45c6-9ed9-dd04a5fd458c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:41.660711 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:41.660685    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-kube-api-access-v9bnz" pod "ca219c8b-7690-45c6-9ed9-dd04a5fd458c" (UID: "ca219c8b-7690-45c6-9ed9-dd04a5fd458c"). InnerVolumeSpecName "kube-api-access-v9bnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:41.759620 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:41.759582    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9bnz\" (UniqueName: \"kubernetes.io/projected/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-kube-api-access-v9bnz\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.759620 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:41.759613    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.759620 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:41.759626    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca219c8b-7690-45c6-9ed9-dd04a5fd458c-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:42.411018 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:42.410983    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9" event={"ID":"ca219c8b-7690-45c6-9ed9-dd04a5fd458c","Type":"ContainerDied","Data":"4b970f474b5760709ae965d8293340f5fdfbea2e61aa851bd709b6e1d8c10939"}
Oct 30 19:45:42.411212 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:42.411067    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b970f474b5760709ae965d8293340f5fdfbea2e61aa851bd709b6e1d8c10939"
Oct 30 19:45:42.411212 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:42.410997    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"
Oct 30 19:45:45.242541 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.242506    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"]
Oct 30 19:45:45.243032 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.242991    6030 cpu_manager.go:414] "containerMap: removing container" podUID="ca219c8b-7690-45c6-9ed9-dd04a5fd458c" containerName="revoke-keys"
Oct 30 19:45:45.243032 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.243003    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca219c8b-7690-45c6-9ed9-dd04a5fd458c" containerName="revoke-keys"
Oct 30 19:45:45.243213 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.243199    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca219c8b-7690-45c6-9ed9-dd04a5fd458c" containerName="revoke-keys"
Oct 30 19:45:45.246432 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.246412    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"
Oct 30 19:45:45.252643 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.252615    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-477fd9c0\""
Oct 30 19:45:45.252783 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.252715    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-477fd9c0-dockercfg-tfpnx\""
Oct 30 19:45:45.259288 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.259256    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"]
Oct 30 19:45:45.275982 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.275912    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"]
Oct 30 19:45:45.279889 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.279841    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"
Oct 30 19:45:45.296990 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.296961    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"]
Oct 30 19:45:45.328613 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.328236    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj\" (UID: \"8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"
Oct 30 19:45:45.330068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.330029    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmql\" (UniqueName: \"kubernetes.io/projected/8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b-kube-api-access-5vmql\") pod \"payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj\" (UID: \"8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"
Oct 30 19:45:45.431159 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.431033    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f98848-567b-4a01-bf8a-97643a717007-plugins-config-volume\") pod \"payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc\" (UID: \"72f98848-567b-4a01-bf8a-97643a717007\") " pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"
Oct 30 19:45:45.431525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.431505    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kkmq\" (UniqueName: \"kubernetes.io/projected/72f98848-567b-4a01-bf8a-97643a717007-kube-api-access-5kkmq\") pod \"payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc\" (UID: \"72f98848-567b-4a01-bf8a-97643a717007\") " pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"
Oct 30 19:45:45.432287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.432260    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmql\" (UniqueName: \"kubernetes.io/projected/8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b-kube-api-access-5vmql\") pod \"payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj\" (UID: \"8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"
Oct 30 19:45:45.432846 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.432807    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj\" (UID: \"8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"
Oct 30 19:45:45.443087 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.436801    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj\" (UID: \"8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"
Oct 30 19:45:45.477638 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.477602    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmql\" (UniqueName: \"kubernetes.io/projected/8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b-kube-api-access-5vmql\") pod \"payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj\" (UID: \"8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"
Oct 30 19:45:45.538241 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.538147    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f98848-567b-4a01-bf8a-97643a717007-plugins-config-volume\") pod \"payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc\" (UID: \"72f98848-567b-4a01-bf8a-97643a717007\") " pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"
Oct 30 19:45:45.538416 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.538285    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kkmq\" (UniqueName: \"kubernetes.io/projected/72f98848-567b-4a01-bf8a-97643a717007-kube-api-access-5kkmq\") pod \"payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc\" (UID: \"72f98848-567b-4a01-bf8a-97643a717007\") " pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"
Oct 30 19:45:45.539309 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.539283    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f98848-567b-4a01-bf8a-97643a717007-plugins-config-volume\") pod \"payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc\" (UID: \"72f98848-567b-4a01-bf8a-97643a717007\") " pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"
Oct 30 19:45:45.551923 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.551899    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kkmq\" (UniqueName: \"kubernetes.io/projected/72f98848-567b-4a01-bf8a-97643a717007-kube-api-access-5kkmq\") pod \"payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc\" (UID: \"72f98848-567b-4a01-bf8a-97643a717007\") " pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"
Oct 30 19:45:45.577333 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.577297    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"
Oct 30 19:45:45.597990 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.597757    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"
Oct 30 19:45:45.896481 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.896455    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"]
Oct 30 19:45:45.897435 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:45:45.897412    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7b9d94_6cc3_454a_b6d5_1545f3c2c26b.slice/crio-b04a87ec32e4183ccac7f58b4b1fdb5a7bd1a5b08f46888187b81e2b215ee68c WatchSource:0}: Error finding container b04a87ec32e4183ccac7f58b4b1fdb5a7bd1a5b08f46888187b81e2b215ee68c: Status 404 returned error can't find the container with id b04a87ec32e4183ccac7f58b4b1fdb5a7bd1a5b08f46888187b81e2b215ee68c
Oct 30 19:45:45.898540 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:45:45.898517    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f98848_567b_4a01_bf8a_97643a717007.slice/crio-469d678f568eb9ea2131ebeff8a992fd6442b8df2a5e12fea526ad3c07103c3d WatchSource:0}: Error finding container 469d678f568eb9ea2131ebeff8a992fd6442b8df2a5e12fea526ad3c07103c3d: Status 404 returned error can't find the container with id 469d678f568eb9ea2131ebeff8a992fd6442b8df2a5e12fea526ad3c07103c3d
Oct 30 19:45:45.899885 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:45.899858    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"]
Oct 30 19:45:46.502371 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:46.502336    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj" event={"ID":"8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b","Type":"ContainerStarted","Data":"43cd9690a07a6f13146a731bedf83d206f5c97692003098f5b25938cff757ccc"}
Oct 30 19:45:46.502818 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:46.502427    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj" event={"ID":"8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b","Type":"ContainerStarted","Data":"b04a87ec32e4183ccac7f58b4b1fdb5a7bd1a5b08f46888187b81e2b215ee68c"}
Oct 30 19:45:46.502818 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:46.502459    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"
Oct 30 19:45:46.504306 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:46.504269    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc" event={"ID":"72f98848-567b-4a01-bf8a-97643a717007","Type":"ContainerStarted","Data":"73a7273bdddc327db8349c21128fecd099f6346ba05a076a831e9c9ae011c6b4"}
Oct 30 19:45:46.504414 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:46.504325    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc" event={"ID":"72f98848-567b-4a01-bf8a-97643a717007","Type":"ContainerStarted","Data":"469d678f568eb9ea2131ebeff8a992fd6442b8df2a5e12fea526ad3c07103c3d"}
Oct 30 19:45:46.504414 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:46.504353    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"
Oct 30 19:45:46.523730 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:46.523682    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj" podStartSLOduration=1.523670309 podStartE2EDuration="1.523670309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:46.522281322 +0000 UTC m=+825.437888179" watchObservedRunningTime="2026-10-30 19:45:46.523670309 +0000 UTC m=+825.439277211"
Oct 30 19:45:46.546099 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:46.546029    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc" podStartSLOduration=1.546013286 podStartE2EDuration="1.546013286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:46.544808582 +0000 UTC m=+825.460415438" watchObservedRunningTime="2026-10-30 19:45:46.546013286 +0000 UTC m=+825.461620143"
Oct 30 19:45:49.917007 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:49.916966    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"]
Oct 30 19:45:49.921854 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:49.921828    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:49.932440 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:49.932419    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7a745c\"/\"distinct-2-w1-7a745c-kserve-self-signed-certs\""
Oct 30 19:45:49.940593 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:49.940565    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"]
Oct 30 19:45:50.031405 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.031371    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-home\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.031590 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.031427    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-model-cache\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.031590 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.031526    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-tls-certs\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.031712 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.031684    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx49x\" (UniqueName: \"kubernetes.io/projected/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-kube-api-access-dx49x\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.031831 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.031808    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-tmp-dir\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.031877 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.031846    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-dshm\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.138069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.132957    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx49x\" (UniqueName: \"kubernetes.io/projected/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-kube-api-access-dx49x\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.138069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.133100    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-tmp-dir\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.138069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.133143    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-dshm\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.138069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.133176    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-home\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.138069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.133240    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-model-cache\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.138069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.133299    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-tls-certs\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.138952 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.138922    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-home\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.139732 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.139710    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-tmp-dir\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.157071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.140125    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-model-cache\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.157316 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.146312    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-dshm\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.182703 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.182627    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-tls-certs\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.221073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.192214    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx49x\" (UniqueName: \"kubernetes.io/projected/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-kube-api-access-dx49x\") pod \"distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") " pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.238072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.235861    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:50.571468 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.571437    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"]
Oct 30 19:45:50.572940 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:45:50.572915    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d1c5f0b_c1e2_4b93_b2d4_be4ece056a09.slice/crio-36de1e800b906cfd4fc2f9eb07422181b6e4fd02b134969f5073386f0c3a5f3a WatchSource:0}: Error finding container 36de1e800b906cfd4fc2f9eb07422181b6e4fd02b134969f5073386f0c3a5f3a: Status 404 returned error can't find the container with id 36de1e800b906cfd4fc2f9eb07422181b6e4fd02b134969f5073386f0c3a5f3a
Oct 30 19:45:50.947446 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.947397    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"]
Oct 30 19:45:50.952569 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.952542    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:45:50.961198 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.961098    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:45:50.961673 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.961643    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-477fd9c0-serving-cert\""
Oct 30 19:45:50.961868 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.961746    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d892r\""
Oct 30 19:45:50.961868 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.961837    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:45:50.961996 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.961898    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:45:50.963953 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:50.963928    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"]
Oct 30 19:45:51.047349 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:51.047318    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd29d816-f679-419a-af25-334751296b51-openshift-service-ca\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz\" (UID: \"cd29d816-f679-419a-af25-334751296b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:45:51.047555 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:51.047363    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd29d816-f679-419a-af25-334751296b51-metrics-certs\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz\" (UID: \"cd29d816-f679-419a-af25-334751296b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:45:51.047555 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:51.047523    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd29d816-f679-419a-af25-334751296b51-maas-api-tls\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz\" (UID: \"cd29d816-f679-419a-af25-334751296b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:45:51.047700 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:51.047561    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbkkj\" (UniqueName: \"kubernetes.io/projected/cd29d816-f679-419a-af25-334751296b51-kube-api-access-gbkkj\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz\" (UID: \"cd29d816-f679-419a-af25-334751296b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:45:51.149918 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:51.149320    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd29d816-f679-419a-af25-334751296b51-openshift-service-ca\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz\" (UID: \"cd29d816-f679-419a-af25-334751296b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:45:51.149918 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:51.149388    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd29d816-f679-419a-af25-334751296b51-metrics-certs\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz\" (UID: \"cd29d816-f679-419a-af25-334751296b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:45:51.149918 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:51.149516    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd29d816-f679-419a-af25-334751296b51-maas-api-tls\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz\" (UID: \"cd29d816-f679-419a-af25-334751296b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:45:51.149918 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:51.149543    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbkkj\" (UniqueName: \"kubernetes.io/projected/cd29d816-f679-419a-af25-334751296b51-kube-api-access-gbkkj\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz\" (UID: \"cd29d816-f679-419a-af25-334751296b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:45:51.157088 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:51.153217    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd29d816-f679-419a-af25-334751296b51-openshift-service-ca\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz\" (UID: \"cd29d816-f679-419a-af25-334751296b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:45:51.157088 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:45:51.153692    6030 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-serving-cert: secret "maas-api-e2e-mt-477fd9c0-serving-cert" not found
Oct 30 19:45:51.157088 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:45:51.153779    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd29d816-f679-419a-af25-334751296b51-maas-api-tls podName:cd29d816-f679-419a-af25-334751296b51 nodeName:}" failed. No retries permitted until 2026-10-30 19:45:51.653759612 +0000 UTC m=+830.569366466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd29d816-f679-419a-af25-334751296b51-maas-api-tls") pod "maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz" (UID: "cd29d816-f679-419a-af25-334751296b51") : secret "maas-api-e2e-mt-477fd9c0-serving-cert" not found
Oct 30 19:45:51.160671 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:51.160644    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd29d816-f679-419a-af25-334751296b51-metrics-certs\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz\" (UID: \"cd29d816-f679-419a-af25-334751296b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:45:51.174115 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:51.169846    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbkkj\" (UniqueName: \"kubernetes.io/projected/cd29d816-f679-419a-af25-334751296b51-kube-api-access-gbkkj\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz\" (UID: \"cd29d816-f679-419a-af25-334751296b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:45:51.545808 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:51.545772    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r" event={"ID":"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09","Type":"ContainerStarted","Data":"b1c8ae9aa337bc68cda478d3c298dc28ef5c7fbd13789f9b2604f083b7f82712"}
Oct 30 19:45:51.545991 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:51.545817    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r" event={"ID":"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09","Type":"ContainerStarted","Data":"36de1e800b906cfd4fc2f9eb07422181b6e4fd02b134969f5073386f0c3a5f3a"}
Oct 30 19:45:51.582071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:51.576817    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r" podStartSLOduration=2.576794975 podStartE2EDuration="2.576794975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:51.568715611 +0000 UTC m=+830.484322462" watchObservedRunningTime="2026-10-30 19:45:51.576794975 +0000 UTC m=+830.492401832"
Oct 30 19:45:51.661083 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:51.657817    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd29d816-f679-419a-af25-334751296b51-maas-api-tls\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz\" (UID: \"cd29d816-f679-419a-af25-334751296b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:45:51.685089 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:51.683515    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd29d816-f679-419a-af25-334751296b51-maas-api-tls\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz\" (UID: \"cd29d816-f679-419a-af25-334751296b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:45:51.870948 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:51.870921    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:45:52.178565 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:52.178535    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"]
Oct 30 19:45:52.180460 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:45:52.180431    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd29d816_f679_419a_af25_334751296b51.slice/crio-8ae9b9cf54085aa18357ddb1d37760e7e8411bb6b9a125f848f6c30dc0f3b8bb WatchSource:0}: Error finding container 8ae9b9cf54085aa18357ddb1d37760e7e8411bb6b9a125f848f6c30dc0f3b8bb: Status 404 returned error can't find the container with id 8ae9b9cf54085aa18357ddb1d37760e7e8411bb6b9a125f848f6c30dc0f3b8bb
Oct 30 19:45:52.181969 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:52.181950    6030 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:45:52.236948 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:52.236918    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:45:52.554979 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:52.554896    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz" event={"ID":"cd29d816-f679-419a-af25-334751296b51","Type":"ContainerStarted","Data":"8ae9b9cf54085aa18357ddb1d37760e7e8411bb6b9a125f848f6c30dc0f3b8bb"}
Oct 30 19:45:53.686069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:53.683435    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz" event={"ID":"cd29d816-f679-419a-af25-334751296b51","Type":"ContainerStarted","Data":"1cc45bd08c414e018962f03657b88f309448d4a88177b66fc6f340b747578389"}
Oct 30 19:45:53.686069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:53.683561    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:45:54.994212 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:54.994162    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:45:55.143209 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:55.143165    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 19:45:55.145375 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:55.145345    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 19:45:55.215526 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:55.215479    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:55.249010 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:55.248915    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:45:55.315070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:55.314138    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:55.440068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:55.439885    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:55.641139 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:55.639156    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:55.651071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:55.647125    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:55.681071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:55.674173    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:55.721071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:55.718209    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:55.997306 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:55.997134    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:55.998471 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:55.998434    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:45:56.065897 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:56.062148    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz" podStartSLOduration=5.541147035 podStartE2EDuration="6.062116076s" totalImagesPullingTime="520.969041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:53.728229021 +0000 UTC m=+832.643835879" watchObservedRunningTime="2026-10-30 19:45:56.062116076 +0000 UTC m=+834.977722934"
Oct 30 19:45:56.261520 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:56.261432    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:45:57.520221 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:57.520190    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"
Oct 30 19:45:57.520672 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:57.520271    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"
Oct 30 19:45:59.774430 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:45:59.774403    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:46:00.237091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:00.237038    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:46:00.249522 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:00.249494    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:46:00.827497 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:00.827462    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:46:01.952578 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:01.952538    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"]
Oct 30 19:46:01.958164 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:01.958141    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"
Oct 30 19:46:01.965877 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:01.965849    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"]
Oct 30 19:46:01.967563 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:01.967524    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6ac47d\""
Oct 30 19:46:01.967998 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:01.967978    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6ac47d-dockercfg-7g7ss\""
Oct 30 19:46:02.051007 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:02.050965    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7hnn\" (UniqueName: \"kubernetes.io/projected/85ca69dd-2084-4fd2-b130-81a8c451b5d6-kube-api-access-q7hnn\") pod \"payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx\" (UID: \"85ca69dd-2084-4fd2-b130-81a8c451b5d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"
Oct 30 19:46:02.051240 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:02.051056    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ca69dd-2084-4fd2-b130-81a8c451b5d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx\" (UID: \"85ca69dd-2084-4fd2-b130-81a8c451b5d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"
Oct 30 19:46:02.156226 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:02.154783    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7hnn\" (UniqueName: \"kubernetes.io/projected/85ca69dd-2084-4fd2-b130-81a8c451b5d6-kube-api-access-q7hnn\") pod \"payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx\" (UID: \"85ca69dd-2084-4fd2-b130-81a8c451b5d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"
Oct 30 19:46:02.156226 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:02.154861    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ca69dd-2084-4fd2-b130-81a8c451b5d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx\" (UID: \"85ca69dd-2084-4fd2-b130-81a8c451b5d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"
Oct 30 19:46:02.156226 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:02.155872    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ca69dd-2084-4fd2-b130-81a8c451b5d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx\" (UID: \"85ca69dd-2084-4fd2-b130-81a8c451b5d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"
Oct 30 19:46:02.182098 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:02.181554    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7hnn\" (UniqueName: \"kubernetes.io/projected/85ca69dd-2084-4fd2-b130-81a8c451b5d6-kube-api-access-q7hnn\") pod \"payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx\" (UID: \"85ca69dd-2084-4fd2-b130-81a8c451b5d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"
Oct 30 19:46:02.274328 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:02.274240    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"
Oct 30 19:46:02.566640 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:02.566609    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"]
Oct 30 19:46:02.567118 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:46:02.567084    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ca69dd_2084_4fd2_b130_81a8c451b5d6.slice/crio-5b9082796feb8dddfb15a112ec5147714e8ee790bafb88aae032970eba413719 WatchSource:0}: Error finding container 5b9082796feb8dddfb15a112ec5147714e8ee790bafb88aae032970eba413719: Status 404 returned error can't find the container with id 5b9082796feb8dddfb15a112ec5147714e8ee790bafb88aae032970eba413719
Oct 30 19:46:02.832607 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:02.832510    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx" event={"ID":"85ca69dd-2084-4fd2-b130-81a8c451b5d6","Type":"ContainerStarted","Data":"be1900a2f8d667301978a1698a1644aef22922c3471c523bb58cb718bffa9677"}
Oct 30 19:46:02.832607 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:02.832587    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"
Oct 30 19:46:02.832887 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:02.832612    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx" event={"ID":"85ca69dd-2084-4fd2-b130-81a8c451b5d6","Type":"ContainerStarted","Data":"5b9082796feb8dddfb15a112ec5147714e8ee790bafb88aae032970eba413719"}
Oct 30 19:46:02.853644 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:02.853593    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx" podStartSLOduration=1.8535777690000002 podStartE2EDuration="1.853577769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:02.853320557 +0000 UTC m=+841.768927415" watchObservedRunningTime="2026-10-30 19:46:02.853577769 +0000 UTC m=+841.769184625"
Oct 30 19:46:04.289819 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:46:04.289779    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:46:07.807101 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:07.807032    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"]
Oct 30 19:46:07.811502 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:07.811479    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:07.821383 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:07.821358    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-477fd9c0\"/\"openshift-service-ca.crt\""
Oct 30 19:46:07.821690 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:07.821672    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-477fd9c0\"/\"default-dockercfg-tmww4\""
Oct 30 19:46:07.821887 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:07.821873    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-477fd9c0\"/\"kube-root-ca.crt\""
Oct 30 19:46:07.821987 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:07.821407    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-477fd9c0\"/\"e2e-lifecycle-model-477fd9c0-kserve-self-signed-certs\""
Oct 30 19:46:07.824728 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:07.824707    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"]
Oct 30 19:46:07.934479 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:07.934441    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-home\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:07.934684 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:07.934489    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-model-cache\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:07.934684 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:07.934657    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-dshm\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:07.934835 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:07.934800    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-tmp-dir\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:07.934892 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:07.934843    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34dbd6be-7d09-4527-b23d-d3bc899a510d-tls-certs\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:07.934892 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:07.934886    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs9wj\" (UniqueName: \"kubernetes.io/projected/34dbd6be-7d09-4527-b23d-d3bc899a510d-kube-api-access-fs9wj\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:08.036361 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:08.036324    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-tmp-dir\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:08.036547 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:08.036401    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34dbd6be-7d09-4527-b23d-d3bc899a510d-tls-certs\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:08.036547 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:08.036466    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs9wj\" (UniqueName: \"kubernetes.io/projected/34dbd6be-7d09-4527-b23d-d3bc899a510d-kube-api-access-fs9wj\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:08.036547 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:08.036538    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-home\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:08.036711 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:08.036562    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-model-cache\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:08.036711 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:08.036656    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-dshm\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:08.036824 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:08.036789    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-tmp-dir\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:08.037083 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:08.037035    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-model-cache\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:08.037392 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:08.037153    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-home\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:08.039255 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:08.039230    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34dbd6be-7d09-4527-b23d-d3bc899a510d-tls-certs\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:08.039419 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:08.039400    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-dshm\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:08.056085 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:08.051154    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs9wj\" (UniqueName: \"kubernetes.io/projected/34dbd6be-7d09-4527-b23d-d3bc899a510d-kube-api-access-fs9wj\") pod \"e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") " pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:08.136073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:08.133422    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:08.495581 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:08.495552    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"]
Oct 30 19:46:08.496266 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:46:08.496240    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34dbd6be_7d09_4527_b23d_d3bc899a510d.slice/crio-0bb34b58414279eed862db66b8a7241a96f5532e01baf3b16fc8e7d1249aafdc WatchSource:0}: Error finding container 0bb34b58414279eed862db66b8a7241a96f5532e01baf3b16fc8e7d1249aafdc: Status 404 returned error can't find the container with id 0bb34b58414279eed862db66b8a7241a96f5532e01baf3b16fc8e7d1249aafdc
Oct 30 19:46:08.877920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:08.877886    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm" event={"ID":"34dbd6be-7d09-4527-b23d-d3bc899a510d","Type":"ContainerStarted","Data":"2359d25006fac502a0e379052c56881547cd013b3a38935460b75b9f70af55c0"}
Oct 30 19:46:08.878327 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:08.877935    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm" event={"ID":"34dbd6be-7d09-4527-b23d-d3bc899a510d","Type":"ContainerStarted","Data":"0bb34b58414279eed862db66b8a7241a96f5532e01baf3b16fc8e7d1249aafdc"}
Oct 30 19:46:08.903411 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:08.903353    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm" podStartSLOduration=1.903338299 podStartE2EDuration="1.903338299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:08.899537239 +0000 UTC m=+847.815144097" watchObservedRunningTime="2026-10-30 19:46:08.903338299 +0000 UTC m=+847.818945154"
Oct 30 19:46:09.134216 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:09.134137    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:13.842831 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:13.842801    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"
Oct 30 19:46:15.225181 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.225146    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"]
Oct 30 19:46:15.229690 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.229660    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.244578 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.244551    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"]
Oct 30 19:46:15.373707 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.373661    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.373858 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.373715    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.373919 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.373891    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.373983 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.373964    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.374150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.374124    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.374288 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.374252    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.374343 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.374320    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.374475 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.374456    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf4j9\" (UniqueName: \"kubernetes.io/projected/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-kube-api-access-nf4j9\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.374524 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.374508    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.475942 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.475843    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4j9\" (UniqueName: \"kubernetes.io/projected/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-kube-api-access-nf4j9\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.475942 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.475908    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.476221 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.475953    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.476221 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.475984    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.476221 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.476095    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.476221 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.476160    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.476221 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.476219    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.476479 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.476291    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.476479 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.476348    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.476479 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.476462    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.476635 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.476517    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.476635 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.476580    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.476748 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.476728    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.476790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.476733    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.478675 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.478653    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.478848 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.478833    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.489549 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.489524    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.489912 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.489890    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf4j9\" (UniqueName: \"kubernetes.io/projected/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-kube-api-access-nf4j9\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.548929 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.548896    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:15.834456 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.834434    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"]
Oct 30 19:46:15.834965 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:46:15.834944    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a121fcd_2a6f_4d1e_9625_d8ccf6868cbd.slice/crio-c774be4f3ca57522148562a5aa5998a17b2343162ee84082ea9a166e643ec00e WatchSource:0}: Error finding container c774be4f3ca57522148562a5aa5998a17b2343162ee84082ea9a166e643ec00e: Status 404 returned error can't find the container with id c774be4f3ca57522148562a5aa5998a17b2343162ee84082ea9a166e643ec00e
Oct 30 19:46:15.837681 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.837648    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:46:15.837763 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.837713    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:46:15.837763 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.837740    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:46:15.967070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.966314    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt" event={"ID":"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd","Type":"ContainerStarted","Data":"0d28a11e3f821105935c0313cc69ae83e8d43578d6b7bb86c66fd22fcbb6fc6d"}
Oct 30 19:46:15.967070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.966377    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt" event={"ID":"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd","Type":"ContainerStarted","Data":"c774be4f3ca57522148562a5aa5998a17b2343162ee84082ea9a166e643ec00e"}
Oct 30 19:46:15.995112 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:15.995032    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt" podStartSLOduration=0.995017173 podStartE2EDuration="995.017173ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:15.991909454 +0000 UTC m=+854.907516321" watchObservedRunningTime="2026-10-30 19:46:15.995017173 +0000 UTC m=+854.910624029"
Oct 30 19:46:16.553160 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:16.553121    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:16.554157 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:16.554124    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:46:16.554321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:16.554297    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt" podUID="3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 19:46:17.549911 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:17.549874    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:46:17.550136 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:17.549929    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt" podUID="3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 19:46:18.134589 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:18.134560    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:18.147216 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:18.147190    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:18.562526 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:18.562439    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:46:18.563251 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:18.563214    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt" podUID="3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 19:46:19.014919 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:19.014891    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:46:19.549290 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:19.549260    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:46:19.549685 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:19.549312    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt" podUID="3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 19:46:20.550223 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:20.550195    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:46:20.550625 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:20.550241    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt" podUID="3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 19:46:21.549445 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:21.549409    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:46:21.549617 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:21.549480    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt" podUID="3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 19:46:22.549629 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:22.549591    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:46:22.550385 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:22.550346    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt" podUID="3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 19:46:24.557245 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:24.557203    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:46:24.557730 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:24.557707    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt" podUID="3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:46:25.599189 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:25.599140    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:46:25.601638 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:25.599215    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt" podUID="3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:46:25.922423 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:25.922392    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:25.923957 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:25.923908    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:25.939731 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:25.938338    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:46:34.288791 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:46:34.288753    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:46:54.456290 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:54.456251    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"]
Oct 30 19:46:54.460291 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:54.460273    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"
Oct 30 19:46:54.466622 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:54.466314    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:46:54.466796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:54.466330    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 19:46:54.469931 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:54.469909    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"]
Oct 30 19:46:54.542688 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:54.542650    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvt7t\" (UniqueName: \"kubernetes.io/projected/93727270-f8e9-4880-8cac-7aa7a5f89f9f-kube-api-access-xvt7t\") pod \"maas-api-revoke-sub-4daf28a3-r2nqz\" (UID: \"93727270-f8e9-4880-8cac-7aa7a5f89f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"
Oct 30 19:46:54.542878 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:54.542846    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93727270-f8e9-4880-8cac-7aa7a5f89f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4daf28a3-r2nqz\" (UID: \"93727270-f8e9-4880-8cac-7aa7a5f89f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"
Oct 30 19:46:54.542933 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:54.542897    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93727270-f8e9-4880-8cac-7aa7a5f89f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4daf28a3-r2nqz\" (UID: \"93727270-f8e9-4880-8cac-7aa7a5f89f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"
Oct 30 19:46:54.643995 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:54.643963    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvt7t\" (UniqueName: \"kubernetes.io/projected/93727270-f8e9-4880-8cac-7aa7a5f89f9f-kube-api-access-xvt7t\") pod \"maas-api-revoke-sub-4daf28a3-r2nqz\" (UID: \"93727270-f8e9-4880-8cac-7aa7a5f89f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"
Oct 30 19:46:54.644287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:54.644116    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93727270-f8e9-4880-8cac-7aa7a5f89f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4daf28a3-r2nqz\" (UID: \"93727270-f8e9-4880-8cac-7aa7a5f89f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"
Oct 30 19:46:54.644287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:54.644153    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93727270-f8e9-4880-8cac-7aa7a5f89f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4daf28a3-r2nqz\" (UID: \"93727270-f8e9-4880-8cac-7aa7a5f89f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"
Oct 30 19:46:54.644714 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:54.644688    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93727270-f8e9-4880-8cac-7aa7a5f89f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4daf28a3-r2nqz\" (UID: \"93727270-f8e9-4880-8cac-7aa7a5f89f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"
Oct 30 19:46:54.646602 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:54.646581    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93727270-f8e9-4880-8cac-7aa7a5f89f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4daf28a3-r2nqz\" (UID: \"93727270-f8e9-4880-8cac-7aa7a5f89f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"
Oct 30 19:46:54.657694 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:54.657671    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvt7t\" (UniqueName: \"kubernetes.io/projected/93727270-f8e9-4880-8cac-7aa7a5f89f9f-kube-api-access-xvt7t\") pod \"maas-api-revoke-sub-4daf28a3-r2nqz\" (UID: \"93727270-f8e9-4880-8cac-7aa7a5f89f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"
Oct 30 19:46:54.776732 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:54.776658    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"
Oct 30 19:46:55.059329 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:55.059299    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"]
Oct 30 19:46:55.059846 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:46:55.059824    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93727270_f8e9_4880_8cac_7aa7a5f89f9f.slice/crio-e184ca79c76fb2381b6201c2906a308df702bf31a630eac1a97c3a073a32c1ce WatchSource:0}: Error finding container e184ca79c76fb2381b6201c2906a308df702bf31a630eac1a97c3a073a32c1ce: Status 404 returned error can't find the container with id e184ca79c76fb2381b6201c2906a308df702bf31a630eac1a97c3a073a32c1ce
Oct 30 19:46:55.387024 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:55.386997    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz" event={"ID":"93727270-f8e9-4880-8cac-7aa7a5f89f9f","Type":"ContainerStarted","Data":"e184ca79c76fb2381b6201c2906a308df702bf31a630eac1a97c3a073a32c1ce"}
Oct 30 19:46:56.395143 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:56.395111    6030 generic.go:356] "Generic (PLEG): container finished" podID="93727270-f8e9-4880-8cac-7aa7a5f89f9f" containerID="809973c7065d7cad9f9acbc6f6ebb8218a13b360559aba56c92688f992f9eaa0" exitCode=0
Oct 30 19:46:56.395540 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:56.395143    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz" event={"ID":"93727270-f8e9-4880-8cac-7aa7a5f89f9f","Type":"ContainerDied","Data":"809973c7065d7cad9f9acbc6f6ebb8218a13b360559aba56c92688f992f9eaa0"}
Oct 30 19:46:57.546603 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:57.546580    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"
Oct 30 19:46:57.594332 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:57.594301    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93727270-f8e9-4880-8cac-7aa7a5f89f9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93727270-f8e9-4880-8cac-7aa7a5f89f9f-maas-api-openshift-ca\") pod \"93727270-f8e9-4880-8cac-7aa7a5f89f9f\" (UID: \"93727270-f8e9-4880-8cac-7aa7a5f89f9f\") "
Oct 30 19:46:57.594528 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:57.594355    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93727270-f8e9-4880-8cac-7aa7a5f89f9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93727270-f8e9-4880-8cac-7aa7a5f89f9f-maas-api-xks-ca\") pod \"93727270-f8e9-4880-8cac-7aa7a5f89f9f\" (UID: \"93727270-f8e9-4880-8cac-7aa7a5f89f9f\") "
Oct 30 19:46:57.594528 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:57.594397    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93727270-f8e9-4880-8cac-7aa7a5f89f9f-kube-api-access-xvt7t\" (UniqueName: \"kubernetes.io/projected/93727270-f8e9-4880-8cac-7aa7a5f89f9f-kube-api-access-xvt7t\") pod \"93727270-f8e9-4880-8cac-7aa7a5f89f9f\" (UID: \"93727270-f8e9-4880-8cac-7aa7a5f89f9f\") "
Oct 30 19:46:57.594727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:57.594697    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93727270-f8e9-4880-8cac-7aa7a5f89f9f-maas-api-openshift-ca" pod "93727270-f8e9-4880-8cac-7aa7a5f89f9f" (UID: "93727270-f8e9-4880-8cac-7aa7a5f89f9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:57.594868 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:57.594850    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93727270-f8e9-4880-8cac-7aa7a5f89f9f-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:57.597253 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:57.597227    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93727270-f8e9-4880-8cac-7aa7a5f89f9f-maas-api-xks-ca" pod "93727270-f8e9-4880-8cac-7aa7a5f89f9f" (UID: "93727270-f8e9-4880-8cac-7aa7a5f89f9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:57.601210 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:57.601179    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93727270-f8e9-4880-8cac-7aa7a5f89f9f-kube-api-access-xvt7t" pod "93727270-f8e9-4880-8cac-7aa7a5f89f9f" (UID: "93727270-f8e9-4880-8cac-7aa7a5f89f9f"). InnerVolumeSpecName "kube-api-access-xvt7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:57.696864 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:57.696771    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93727270-f8e9-4880-8cac-7aa7a5f89f9f-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:57.696864 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:57.696809    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvt7t\" (UniqueName: \"kubernetes.io/projected/93727270-f8e9-4880-8cac-7aa7a5f89f9f-kube-api-access-xvt7t\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:58.410030 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:58.409994    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz" event={"ID":"93727270-f8e9-4880-8cac-7aa7a5f89f9f","Type":"ContainerDied","Data":"e184ca79c76fb2381b6201c2906a308df702bf31a630eac1a97c3a073a32c1ce"}
Oct 30 19:46:58.410216 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:58.410032    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"
Oct 30 19:46:58.410216 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:46:58.410071    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e184ca79c76fb2381b6201c2906a308df702bf31a630eac1a97c3a073a32c1ce"
Oct 30 19:47:03.047021 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:03.046983    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"]
Oct 30 19:47:04.290453 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:47:04.290414    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:47:08.048177 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:08.048137    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx" podUID="85ca69dd-2084-4fd2-b130-81a8c451b5d6" containerName="payload-pre-processing" containerID="cri-o://be1900a2f8d667301978a1698a1644aef22922c3471c523bb58cb718bffa9677" gracePeriod=25
Oct 30 19:47:08.317576 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:08.317519    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"
Oct 30 19:47:08.470348 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:08.470314    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ca69dd-2084-4fd2-b130-81a8c451b5d6-kube-api-access-q7hnn\" (UniqueName: \"kubernetes.io/projected/85ca69dd-2084-4fd2-b130-81a8c451b5d6-kube-api-access-q7hnn\") pod \"85ca69dd-2084-4fd2-b130-81a8c451b5d6\" (UID: \"85ca69dd-2084-4fd2-b130-81a8c451b5d6\") "
Oct 30 19:47:08.470533 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:08.470477    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85ca69dd-2084-4fd2-b130-81a8c451b5d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ca69dd-2084-4fd2-b130-81a8c451b5d6-plugins-config-volume\") pod \"85ca69dd-2084-4fd2-b130-81a8c451b5d6\" (UID: \"85ca69dd-2084-4fd2-b130-81a8c451b5d6\") "
Oct 30 19:47:08.470836 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:08.470811    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ca69dd-2084-4fd2-b130-81a8c451b5d6-plugins-config-volume" pod "85ca69dd-2084-4fd2-b130-81a8c451b5d6" (UID: "85ca69dd-2084-4fd2-b130-81a8c451b5d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:08.471006 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:08.470982    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ca69dd-2084-4fd2-b130-81a8c451b5d6-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.472480 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:08.472464    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ca69dd-2084-4fd2-b130-81a8c451b5d6-kube-api-access-q7hnn" pod "85ca69dd-2084-4fd2-b130-81a8c451b5d6" (UID: "85ca69dd-2084-4fd2-b130-81a8c451b5d6"). InnerVolumeSpecName "kube-api-access-q7hnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:08.490318 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:08.490292    6030 generic.go:356] "Generic (PLEG): container finished" podID="85ca69dd-2084-4fd2-b130-81a8c451b5d6" containerID="be1900a2f8d667301978a1698a1644aef22922c3471c523bb58cb718bffa9677" exitCode=0
Oct 30 19:47:08.490442 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:08.490344    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"
Oct 30 19:47:08.490442 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:08.490374    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx" event={"ID":"85ca69dd-2084-4fd2-b130-81a8c451b5d6","Type":"ContainerDied","Data":"be1900a2f8d667301978a1698a1644aef22922c3471c523bb58cb718bffa9677"}
Oct 30 19:47:08.490442 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:08.490425    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx" event={"ID":"85ca69dd-2084-4fd2-b130-81a8c451b5d6","Type":"ContainerDied","Data":"5b9082796feb8dddfb15a112ec5147714e8ee790bafb88aae032970eba413719"}
Oct 30 19:47:08.490594 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:08.490450    6030 scope.go:122] "RemoveContainer" containerID="be1900a2f8d667301978a1698a1644aef22922c3471c523bb58cb718bffa9677"
Oct 30 19:47:08.500341 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:08.500322    6030 scope.go:122] "RemoveContainer" containerID="be1900a2f8d667301978a1698a1644aef22922c3471c523bb58cb718bffa9677"
Oct 30 19:47:08.500597 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:47:08.500579    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1900a2f8d667301978a1698a1644aef22922c3471c523bb58cb718bffa9677\": container with ID starting with be1900a2f8d667301978a1698a1644aef22922c3471c523bb58cb718bffa9677 not found: ID does not exist" containerID="be1900a2f8d667301978a1698a1644aef22922c3471c523bb58cb718bffa9677"
Oct 30 19:47:08.500645 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:08.500606    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1900a2f8d667301978a1698a1644aef22922c3471c523bb58cb718bffa9677"} err="failed to get container status \"be1900a2f8d667301978a1698a1644aef22922c3471c523bb58cb718bffa9677\": rpc error: code = NotFound desc = could not find container \"be1900a2f8d667301978a1698a1644aef22922c3471c523bb58cb718bffa9677\": container with ID starting with be1900a2f8d667301978a1698a1644aef22922c3471c523bb58cb718bffa9677 not found: ID does not exist"
Oct 30 19:47:08.520713 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:08.520688    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"]
Oct 30 19:47:08.523473 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:08.523453    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ac47d-7b459856ff-w6ttx"]
Oct 30 19:47:08.572213 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:08.572156    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7hnn\" (UniqueName: \"kubernetes.io/projected/85ca69dd-2084-4fd2-b130-81a8c451b5d6-kube-api-access-q7hnn\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:09.201112 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:09.201083    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85ca69dd-2084-4fd2-b130-81a8c451b5d6" path="/var/lib/kubelet/pods/85ca69dd-2084-4fd2-b130-81a8c451b5d6/volumes"
Oct 30 19:47:13.058723 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:13.058681    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"]
Oct 30 19:47:13.061469 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:13.061447    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb905a7b-mnj7w"]
Oct 30 19:47:13.201065 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:13.201014    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aea1208b-a0f4-4882-9da5-7e85331975b3" path="/var/lib/kubelet/pods/aea1208b-a0f4-4882-9da5-7e85331975b3/volumes"
Oct 30 19:47:22.094135 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:22.094099    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"]
Oct 30 19:47:22.108108 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:22.108072    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebcddb67-qm6rs"]
Oct 30 19:47:23.201654 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:23.201619    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b1376e-f1d9-4d02-8371-cefc8896a416" path="/var/lib/kubelet/pods/a8b1376e-f1d9-4d02-8371-cefc8896a416/volumes"
Oct 30 19:47:25.560671 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:25.560641    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"]
Oct 30 19:47:25.573601 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:25.573569    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"]
Oct 30 19:47:30.561266 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:30.560996    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc" podUID="72f98848-567b-4a01-bf8a-97643a717007" containerName="payload-processing" containerID="cri-o://73a7273bdddc327db8349c21128fecd099f6346ba05a076a831e9c9ae011c6b4" gracePeriod=25
Oct 30 19:47:30.574124 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:30.574087    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj" podUID="8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b" containerName="payload-pre-processing" containerID="cri-o://43cd9690a07a6f13146a731bedf83d206f5c97692003098f5b25938cff757ccc" gracePeriod=25
Oct 30 19:47:34.291287 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:47:34.291245    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:47:36.053168 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:36.053129    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"]
Oct 30 19:47:36.057790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:36.057762    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e11f28b-dsfrf"]
Oct 30 19:47:37.200536 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:37.200504    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76b58bf3-e970-4384-9f03-faacc5c95f4c" path="/var/lib/kubelet/pods/76b58bf3-e970-4384-9f03-faacc5c95f4c/volumes"
Oct 30 19:47:37.518174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:37.518089    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc" podUID="72f98848-567b-4a01-bf8a-97643a717007" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:47:37.518174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:37.518095    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj" podUID="8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:47:38.517661 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:38.517627    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"]
Oct 30 19:47:38.518182 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:38.517912    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz" podUID="cd29d816-f679-419a-af25-334751296b51" containerName="maas-api" containerID="cri-o://1cc45bd08c414e018962f03657b88f309448d4a88177b66fc6f340b747578389" gracePeriod=30
Oct 30 19:47:38.894150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:38.884067    6030 generic.go:356] "Generic (PLEG): container finished" podID="cd29d816-f679-419a-af25-334751296b51" containerID="1cc45bd08c414e018962f03657b88f309448d4a88177b66fc6f340b747578389" exitCode=0
Oct 30 19:47:38.894150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:38.884306    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz" event={"ID":"cd29d816-f679-419a-af25-334751296b51","Type":"ContainerDied","Data":"1cc45bd08c414e018962f03657b88f309448d4a88177b66fc6f340b747578389"}
Oct 30 19:47:39.007367 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.007332    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:47:39.095458 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.095427    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd29d816-f679-419a-af25-334751296b51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd29d816-f679-419a-af25-334751296b51-metrics-certs\") pod \"cd29d816-f679-419a-af25-334751296b51\" (UID: \"cd29d816-f679-419a-af25-334751296b51\") "
Oct 30 19:47:39.095629 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.095526    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd29d816-f679-419a-af25-334751296b51-kube-api-access-gbkkj\" (UniqueName: \"kubernetes.io/projected/cd29d816-f679-419a-af25-334751296b51-kube-api-access-gbkkj\") pod \"cd29d816-f679-419a-af25-334751296b51\" (UID: \"cd29d816-f679-419a-af25-334751296b51\") "
Oct 30 19:47:39.095629 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.095592    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd29d816-f679-419a-af25-334751296b51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd29d816-f679-419a-af25-334751296b51-openshift-service-ca\") pod \"cd29d816-f679-419a-af25-334751296b51\" (UID: \"cd29d816-f679-419a-af25-334751296b51\") "
Oct 30 19:47:39.096188 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.095640    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd29d816-f679-419a-af25-334751296b51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd29d816-f679-419a-af25-334751296b51-maas-api-tls\") pod \"cd29d816-f679-419a-af25-334751296b51\" (UID: \"cd29d816-f679-419a-af25-334751296b51\") "
Oct 30 19:47:39.096188 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.095962    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd29d816-f679-419a-af25-334751296b51-openshift-service-ca" pod "cd29d816-f679-419a-af25-334751296b51" (UID: "cd29d816-f679-419a-af25-334751296b51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:39.096188 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.096140    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd29d816-f679-419a-af25-334751296b51-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:39.097747 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.097718    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd29d816-f679-419a-af25-334751296b51-metrics-certs" pod "cd29d816-f679-419a-af25-334751296b51" (UID: "cd29d816-f679-419a-af25-334751296b51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:39.097883 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.097823    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd29d816-f679-419a-af25-334751296b51-kube-api-access-gbkkj" pod "cd29d816-f679-419a-af25-334751296b51" (UID: "cd29d816-f679-419a-af25-334751296b51"). InnerVolumeSpecName "kube-api-access-gbkkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:39.097883 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.097835    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd29d816-f679-419a-af25-334751296b51-maas-api-tls" pod "cd29d816-f679-419a-af25-334751296b51" (UID: "cd29d816-f679-419a-af25-334751296b51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:39.201534 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.201397    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd29d816-f679-419a-af25-334751296b51-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:39.201534 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.201431    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd29d816-f679-419a-af25-334751296b51-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:39.201534 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.201444    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbkkj\" (UniqueName: \"kubernetes.io/projected/cd29d816-f679-419a-af25-334751296b51-kube-api-access-gbkkj\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:39.893297 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.893267    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"
Oct 30 19:47:39.893297 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.893279    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz" event={"ID":"cd29d816-f679-419a-af25-334751296b51","Type":"ContainerDied","Data":"8ae9b9cf54085aa18357ddb1d37760e7e8411bb6b9a125f848f6c30dc0f3b8bb"}
Oct 30 19:47:39.893297 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.893340    6030 scope.go:122] "RemoveContainer" containerID="1cc45bd08c414e018962f03657b88f309448d4a88177b66fc6f340b747578389"
Oct 30 19:47:39.915866 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.915834    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"]
Oct 30 19:47:39.922634 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:39.922606    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-gkknz"]
Oct 30 19:47:41.202693 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:41.202663    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd29d816-f679-419a-af25-334751296b51" path="/var/lib/kubelet/pods/cd29d816-f679-419a-af25-334751296b51/volumes"
Oct 30 19:47:43.064161 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:43.064103    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"]
Oct 30 19:47:43.075224 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:43.075176    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49cf390e-9cf7z"]
Oct 30 19:47:43.207547 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:43.207515    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="357ea7bb-a182-4b5e-8f73-25b23e9370e9" path="/var/lib/kubelet/pods/357ea7bb-a182-4b5e-8f73-25b23e9370e9/volumes"
Oct 30 19:47:45.407197 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.407159    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"]
Oct 30 19:47:45.408285 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.408253    6030 cpu_manager.go:414] "containerMap: removing container" podUID="93727270-f8e9-4880-8cac-7aa7a5f89f9f" containerName="revoke-keys"
Oct 30 19:47:45.408285 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.408279    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93727270-f8e9-4880-8cac-7aa7a5f89f9f" containerName="revoke-keys"
Oct 30 19:47:45.408487 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.408307    6030 cpu_manager.go:414] "containerMap: removing container" podUID="cd29d816-f679-419a-af25-334751296b51" containerName="maas-api"
Oct 30 19:47:45.408487 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.408315    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd29d816-f679-419a-af25-334751296b51" containerName="maas-api"
Oct 30 19:47:45.408487 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.408339    6030 cpu_manager.go:414] "containerMap: removing container" podUID="85ca69dd-2084-4fd2-b130-81a8c451b5d6" containerName="payload-pre-processing"
Oct 30 19:47:45.408487 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.408346    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ca69dd-2084-4fd2-b130-81a8c451b5d6" containerName="payload-pre-processing"
Oct 30 19:47:45.408487 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.408464    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd29d816-f679-419a-af25-334751296b51" containerName="maas-api"
Oct 30 19:47:45.408487 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.408480    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ca69dd-2084-4fd2-b130-81a8c451b5d6" containerName="payload-pre-processing"
Oct 30 19:47:45.408787 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.408494    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="93727270-f8e9-4880-8cac-7aa7a5f89f9f" containerName="revoke-keys"
Oct 30 19:47:45.411753 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.411731    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.425730 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.423917    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"]
Oct 30 19:47:45.537908 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.537877    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.538140 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.537923    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce705f6-f914-463d-9093-2599d0db1db7-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.538140 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.537998    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.538140 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.538102    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx459\" (UniqueName: \"kubernetes.io/projected/cce705f6-f914-463d-9093-2599d0db1db7-kube-api-access-lx459\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.538140 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.538128    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.538362 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.538157    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce705f6-f914-463d-9093-2599d0db1db7-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.538362 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.538238    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.538362 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.538256    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce705f6-f914-463d-9093-2599d0db1db7-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.538362 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.538282    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.639512 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.639475    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.639699 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.639548    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce705f6-f914-463d-9093-2599d0db1db7-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.639699 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.639582    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.639699 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.639642    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx459\" (UniqueName: \"kubernetes.io/projected/cce705f6-f914-463d-9093-2599d0db1db7-kube-api-access-lx459\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.639699 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.639677    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.639904 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.639726    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce705f6-f914-463d-9093-2599d0db1db7-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.639904 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.639757    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.639998 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.639949    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.639998 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.639968    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce705f6-f914-463d-9093-2599d0db1db7-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.640119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.640036    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.640169 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.640132    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.640278 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.640256    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.640359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.640345    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.640680 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.640654    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce705f6-f914-463d-9093-2599d0db1db7-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.642440 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.642415    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.642830 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.642809    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce705f6-f914-463d-9093-2599d0db1db7-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.681882 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.681806    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx459\" (UniqueName: \"kubernetes.io/projected/cce705f6-f914-463d-9093-2599d0db1db7-kube-api-access-lx459\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.699730 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.699654    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce705f6-f914-463d-9093-2599d0db1db7-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:45.731877 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:45.731841    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:46.022563 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:46.022536    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"]
Oct 30 19:47:46.023210 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:47:46.023183    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce705f6_f914_463d_9093_2599d0db1db7.slice/crio-e8bbe82656796da1ba300278be2cbd07073820f485bfd3d334e1f9b29b8d8a7b WatchSource:0}: Error finding container e8bbe82656796da1ba300278be2cbd07073820f485bfd3d334e1f9b29b8d8a7b: Status 404 returned error can't find the container with id e8bbe82656796da1ba300278be2cbd07073820f485bfd3d334e1f9b29b8d8a7b
Oct 30 19:47:46.025652 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:46.025619    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:47:46.025741 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:46.025685    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:47:46.025741 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:46.025713    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:47:47.002108 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:47.002036    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr" event={"ID":"cce705f6-f914-463d-9093-2599d0db1db7","Type":"ContainerStarted","Data":"5de93769953a93248b5d81d78718ca4cce25877baed6353d6995796e735f2e30"}
Oct 30 19:47:47.002508 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:47.002130    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr" event={"ID":"cce705f6-f914-463d-9093-2599d0db1db7","Type":"ContainerStarted","Data":"e8bbe82656796da1ba300278be2cbd07073820f485bfd3d334e1f9b29b8d8a7b"}
Oct 30 19:47:47.045805 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:47.045751    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr" podStartSLOduration=2.045736232 podStartE2EDuration="2.045736232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:47.041308922 +0000 UTC m=+945.956915788" watchObservedRunningTime="2026-10-30 19:47:47.045736232 +0000 UTC m=+945.961343084"
Oct 30 19:47:47.518617 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:47.518579    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc" podUID="72f98848-567b-4a01-bf8a-97643a717007" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:47:47.518810 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:47.518630    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj" podUID="8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:47:47.732672 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:47.732644    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:47.734163 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:47.734138    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 19:47:47.734260 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:47.734192    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr" podUID="cce705f6-f914-463d-9093-2599d0db1db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 19:47:48.741067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:48.740633    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 19:47:48.741067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:48.740710    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr" podUID="cce705f6-f914-463d-9093-2599d0db1db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 19:47:49.733432 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:49.733396    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 19:47:49.733625 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:49.733456    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr" podUID="cce705f6-f914-463d-9093-2599d0db1db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 19:47:50.733671 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:50.733634    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 19:47:50.734166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:50.733703    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr" podUID="cce705f6-f914-463d-9093-2599d0db1db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 19:47:50.954556 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:50.954519    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"
Oct 30 19:47:51.029222 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.029188    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"
Oct 30 19:47:51.032145 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.032114    6030 generic.go:356] "Generic (PLEG): container finished" podID="8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b" containerID="43cd9690a07a6f13146a731bedf83d206f5c97692003098f5b25938cff757ccc" exitCode=0
Oct 30 19:47:51.032269 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.032161    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"
Oct 30 19:47:51.032269 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.032186    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj" event={"ID":"8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b","Type":"ContainerDied","Data":"43cd9690a07a6f13146a731bedf83d206f5c97692003098f5b25938cff757ccc"}
Oct 30 19:47:51.032395 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.032273    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj" event={"ID":"8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b","Type":"ContainerDied","Data":"b04a87ec32e4183ccac7f58b4b1fdb5a7bd1a5b08f46888187b81e2b215ee68c"}
Oct 30 19:47:51.032395 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.032323    6030 scope.go:122] "RemoveContainer" containerID="43cd9690a07a6f13146a731bedf83d206f5c97692003098f5b25938cff757ccc"
Oct 30 19:47:51.033916 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.033894    6030 generic.go:356] "Generic (PLEG): container finished" podID="72f98848-567b-4a01-bf8a-97643a717007" containerID="73a7273bdddc327db8349c21128fecd099f6346ba05a076a831e9c9ae011c6b4" exitCode=0
Oct 30 19:47:51.033983 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.033933    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"
Oct 30 19:47:51.034035 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.033975    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc" event={"ID":"72f98848-567b-4a01-bf8a-97643a717007","Type":"ContainerDied","Data":"73a7273bdddc327db8349c21128fecd099f6346ba05a076a831e9c9ae011c6b4"}
Oct 30 19:47:51.034035 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.034021    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc" event={"ID":"72f98848-567b-4a01-bf8a-97643a717007","Type":"ContainerDied","Data":"469d678f568eb9ea2131ebeff8a992fd6442b8df2a5e12fea526ad3c07103c3d"}
Oct 30 19:47:51.043346 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.043323    6030 scope.go:122] "RemoveContainer" containerID="43cd9690a07a6f13146a731bedf83d206f5c97692003098f5b25938cff757ccc"
Oct 30 19:47:51.043628 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:47:51.043609    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43cd9690a07a6f13146a731bedf83d206f5c97692003098f5b25938cff757ccc\": container with ID starting with 43cd9690a07a6f13146a731bedf83d206f5c97692003098f5b25938cff757ccc not found: ID does not exist" containerID="43cd9690a07a6f13146a731bedf83d206f5c97692003098f5b25938cff757ccc"
Oct 30 19:47:51.043717 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.043644    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43cd9690a07a6f13146a731bedf83d206f5c97692003098f5b25938cff757ccc"} err="failed to get container status \"43cd9690a07a6f13146a731bedf83d206f5c97692003098f5b25938cff757ccc\": rpc error: code = NotFound desc = could not find container \"43cd9690a07a6f13146a731bedf83d206f5c97692003098f5b25938cff757ccc\": container with ID starting with 43cd9690a07a6f13146a731bedf83d206f5c97692003098f5b25938cff757ccc not found: ID does not exist"
Oct 30 19:47:51.043717 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.043668    6030 scope.go:122] "RemoveContainer" containerID="73a7273bdddc327db8349c21128fecd099f6346ba05a076a831e9c9ae011c6b4"
Oct 30 19:47:51.052904 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.052878    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f98848-567b-4a01-bf8a-97643a717007-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f98848-567b-4a01-bf8a-97643a717007-plugins-config-volume\") pod \"72f98848-567b-4a01-bf8a-97643a717007\" (UID: \"72f98848-567b-4a01-bf8a-97643a717007\") "
Oct 30 19:47:51.053030 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.052995    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b-plugins-config-volume\") pod \"8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b\" (UID: \"8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b\") "
Oct 30 19:47:51.053030 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.053027    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f98848-567b-4a01-bf8a-97643a717007-kube-api-access-5kkmq\" (UniqueName: \"kubernetes.io/projected/72f98848-567b-4a01-bf8a-97643a717007-kube-api-access-5kkmq\") pod \"72f98848-567b-4a01-bf8a-97643a717007\" (UID: \"72f98848-567b-4a01-bf8a-97643a717007\") "
Oct 30 19:47:51.053208 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.053074    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b-kube-api-access-5vmql\" (UniqueName: \"kubernetes.io/projected/8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b-kube-api-access-5vmql\") pod \"8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b\" (UID: \"8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b\") "
Oct 30 19:47:51.056173 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.056121    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b-kube-api-access-5vmql" pod "8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b" (UID: "8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b"). InnerVolumeSpecName "kube-api-access-5vmql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:51.056441 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.056416    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f98848-567b-4a01-bf8a-97643a717007-plugins-config-volume" pod "72f98848-567b-4a01-bf8a-97643a717007" (UID: "72f98848-567b-4a01-bf8a-97643a717007"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:51.056564 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.056475    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b-plugins-config-volume" pod "8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b" (UID: "8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:51.057269 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.057190    6030 scope.go:122] "RemoveContainer" containerID="73a7273bdddc327db8349c21128fecd099f6346ba05a076a831e9c9ae011c6b4"
Oct 30 19:47:51.057609 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:47:51.057580    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a7273bdddc327db8349c21128fecd099f6346ba05a076a831e9c9ae011c6b4\": container with ID starting with 73a7273bdddc327db8349c21128fecd099f6346ba05a076a831e9c9ae011c6b4 not found: ID does not exist" containerID="73a7273bdddc327db8349c21128fecd099f6346ba05a076a831e9c9ae011c6b4"
Oct 30 19:47:51.057687 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.057621    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a7273bdddc327db8349c21128fecd099f6346ba05a076a831e9c9ae011c6b4"} err="failed to get container status \"73a7273bdddc327db8349c21128fecd099f6346ba05a076a831e9c9ae011c6b4\": rpc error: code = NotFound desc = could not find container \"73a7273bdddc327db8349c21128fecd099f6346ba05a076a831e9c9ae011c6b4\": container with ID starting with 73a7273bdddc327db8349c21128fecd099f6346ba05a076a831e9c9ae011c6b4 not found: ID does not exist"
Oct 30 19:47:51.058570 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.058547    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f98848-567b-4a01-bf8a-97643a717007-kube-api-access-5kkmq" pod "72f98848-567b-4a01-bf8a-97643a717007" (UID: "72f98848-567b-4a01-bf8a-97643a717007"). InnerVolumeSpecName "kube-api-access-5kkmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:51.154969 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.154893    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f98848-567b-4a01-bf8a-97643a717007-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.154969 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.154923    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.154969 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.154932    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kkmq\" (UniqueName: \"kubernetes.io/projected/72f98848-567b-4a01-bf8a-97643a717007-kube-api-access-5kkmq\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.154969 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.154941    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vmql\" (UniqueName: \"kubernetes.io/projected/8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b-kube-api-access-5vmql\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.356267 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.356237    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"]
Oct 30 19:47:51.361706 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.361680    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-477fd9c0-84cd565b7f-z7bmj"]
Oct 30 19:47:51.398070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.393238    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"]
Oct 30 19:47:51.398649 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.398615    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-m44hc"]
Oct 30 19:47:51.743115 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.732794    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 19:47:51.743115 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.732852    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr" podUID="cce705f6-f914-463d-9093-2599d0db1db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 19:47:51.962323 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.962244    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"]
Oct 30 19:47:51.962820 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.962807    6030 cpu_manager.go:414] "containerMap: removing container" podUID="8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b" containerName="payload-pre-processing"
Oct 30 19:47:51.962860 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.962822    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b" containerName="payload-pre-processing"
Oct 30 19:47:51.962860 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.962833    6030 cpu_manager.go:414] "containerMap: removing container" podUID="72f98848-567b-4a01-bf8a-97643a717007" containerName="payload-processing"
Oct 30 19:47:51.962860 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.962838    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f98848-567b-4a01-bf8a-97643a717007" containerName="payload-processing"
Oct 30 19:47:51.962963 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.962931    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b" containerName="payload-pre-processing"
Oct 30 19:47:51.962963 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.962939    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f98848-567b-4a01-bf8a-97643a717007" containerName="payload-processing"
Oct 30 19:47:51.967752 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.967734    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"
Oct 30 19:47:51.990206 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.990133    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-477fd9c0\""
Oct 30 19:47:51.990206 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.990166    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-477fd9c0-dockercfg-fmnbv\""
Oct 30 19:47:51.994239 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:51.994201    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"]
Oct 30 19:47:52.086093 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:52.069475    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc9886a-2484-42d7-acde-a8724a17d8cb-plugins-config-volume\") pod \"payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln\" (UID: \"4fc9886a-2484-42d7-acde-a8724a17d8cb\") " pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"
Oct 30 19:47:52.086093 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:52.069687    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94pxt\" (UniqueName: \"kubernetes.io/projected/4fc9886a-2484-42d7-acde-a8724a17d8cb-kube-api-access-94pxt\") pod \"payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln\" (UID: \"4fc9886a-2484-42d7-acde-a8724a17d8cb\") " pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"
Oct 30 19:47:52.172080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:52.170833    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc9886a-2484-42d7-acde-a8724a17d8cb-plugins-config-volume\") pod \"payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln\" (UID: \"4fc9886a-2484-42d7-acde-a8724a17d8cb\") " pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"
Oct 30 19:47:52.172080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:52.171076    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94pxt\" (UniqueName: \"kubernetes.io/projected/4fc9886a-2484-42d7-acde-a8724a17d8cb-kube-api-access-94pxt\") pod \"payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln\" (UID: \"4fc9886a-2484-42d7-acde-a8724a17d8cb\") " pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"
Oct 30 19:47:52.172080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:52.171940    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc9886a-2484-42d7-acde-a8724a17d8cb-plugins-config-volume\") pod \"payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln\" (UID: \"4fc9886a-2484-42d7-acde-a8724a17d8cb\") " pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"
Oct 30 19:47:52.195146 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:52.193689    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"]
Oct 30 19:47:52.218625 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:52.199347    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96878bba-g5xwq"]
Oct 30 19:47:52.218625 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:52.203795    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94pxt\" (UniqueName: \"kubernetes.io/projected/4fc9886a-2484-42d7-acde-a8724a17d8cb-kube-api-access-94pxt\") pod \"payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln\" (UID: \"4fc9886a-2484-42d7-acde-a8724a17d8cb\") " pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"
Oct 30 19:47:52.305007 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:52.304973    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"
Oct 30 19:47:52.617225 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:52.617198    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"]
Oct 30 19:47:52.619626 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:47:52.619595    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc9886a_2484_42d7_acde_a8724a17d8cb.slice/crio-a5d2b4c2d52bf16eb4e186354e8f95986a6dc723397582b310b026d668ac8f1b WatchSource:0}: Error finding container a5d2b4c2d52bf16eb4e186354e8f95986a6dc723397582b310b026d668ac8f1b: Status 404 returned error can't find the container with id a5d2b4c2d52bf16eb4e186354e8f95986a6dc723397582b310b026d668ac8f1b
Oct 30 19:47:52.733147 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:52.733106    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 19:47:52.733317 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:52.733162    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr" podUID="cce705f6-f914-463d-9093-2599d0db1db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 19:47:53.058526 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:53.058487    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln" event={"ID":"4fc9886a-2484-42d7-acde-a8724a17d8cb","Type":"ContainerStarted","Data":"feb58bd4087a5dbb55261306dbe4e1e2942f87e491872872b1e50fdfd5c5571b"}
Oct 30 19:47:53.058944 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:53.058550    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln" event={"ID":"4fc9886a-2484-42d7-acde-a8724a17d8cb","Type":"ContainerStarted","Data":"a5d2b4c2d52bf16eb4e186354e8f95986a6dc723397582b310b026d668ac8f1b"}
Oct 30 19:47:53.058944 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:53.058581    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"
Oct 30 19:47:53.217684 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:53.217650    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f98848-567b-4a01-bf8a-97643a717007" path="/var/lib/kubelet/pods/72f98848-567b-4a01-bf8a-97643a717007/volumes"
Oct 30 19:47:53.218233 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:53.218212    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89527289-dff4-4c7e-9eeb-6e981faf1a21" path="/var/lib/kubelet/pods/89527289-dff4-4c7e-9eeb-6e981faf1a21/volumes"
Oct 30 19:47:53.218645 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:53.218627    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b" path="/var/lib/kubelet/pods/8b7b9d94-6cc3-454a-b6d5-1545f3c2c26b/volumes"
Oct 30 19:47:53.737113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:53.737084    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:53.737421 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:53.737384    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:53.738224 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:53.738206    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 19:47:53.775347 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:53.775293    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln" podStartSLOduration=2.775277846 podStartE2EDuration="2.775277846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:53.082195285 +0000 UTC m=+951.997802142" watchObservedRunningTime="2026-10-30 19:47:53.775277846 +0000 UTC m=+952.690884765"
Oct 30 19:47:56.149141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:56.149092    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"]
Oct 30 19:47:56.161107 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:56.158229    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7b31cf5-t6xfr"]
Oct 30 19:47:57.206381 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:57.206337    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f697726-1e6f-4463-b00f-0f4090608380" path="/var/lib/kubelet/pods/0f697726-1e6f-4463-b00f-0f4090608380/volumes"
Oct 30 19:47:59.798072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:59.796157    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:47:59.798072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:47:59.796265    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6" podUID="39855966-df67-4ccf-87fe-c5ec63f95cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:00.452677 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.452641    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"]
Oct 30 19:48:00.457997 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.457968    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:00.469011 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.468446    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"]
Oct 30 19:48:00.470400 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.470300    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d892r\""
Oct 30 19:48:00.470400 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.470320    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:48:00.470400 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.470379    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-477fd9c0-serving-cert\""
Oct 30 19:48:00.470613 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.470415    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:48:00.470613 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.470500    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:48:00.485072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.484532    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e754ad-eb64-4930-9e55-abf3abaca1b3-openshift-service-ca\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj\" (UID: \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:00.485072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.484760    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1e754ad-eb64-4930-9e55-abf3abaca1b3-maas-api-tls\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj\" (UID: \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:00.485072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.484798    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1e754ad-eb64-4930-9e55-abf3abaca1b3-metrics-certs\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj\" (UID: \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:00.485072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.484827    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcmq6\" (UniqueName: \"kubernetes.io/projected/f1e754ad-eb64-4930-9e55-abf3abaca1b3-kube-api-access-xcmq6\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj\" (UID: \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:00.586613 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.586549    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1e754ad-eb64-4930-9e55-abf3abaca1b3-maas-api-tls\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj\" (UID: \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:00.586811 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.586629    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1e754ad-eb64-4930-9e55-abf3abaca1b3-metrics-certs\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj\" (UID: \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:00.586811 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.586660    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcmq6\" (UniqueName: \"kubernetes.io/projected/f1e754ad-eb64-4930-9e55-abf3abaca1b3-kube-api-access-xcmq6\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj\" (UID: \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:00.586811 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:48:00.586713    6030 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-serving-cert: secret "maas-api-e2e-mt-477fd9c0-serving-cert" not found
Oct 30 19:48:00.586811 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:48:00.586787    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1e754ad-eb64-4930-9e55-abf3abaca1b3-maas-api-tls podName:f1e754ad-eb64-4930-9e55-abf3abaca1b3 nodeName:}" failed. No retries permitted until 2026-10-30 19:48:01.086767158 +0000 UTC m=+960.002374007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f1e754ad-eb64-4930-9e55-abf3abaca1b3-maas-api-tls") pod "maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj" (UID: "f1e754ad-eb64-4930-9e55-abf3abaca1b3") : secret "maas-api-e2e-mt-477fd9c0-serving-cert" not found
Oct 30 19:48:00.587022 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.586859    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e754ad-eb64-4930-9e55-abf3abaca1b3-openshift-service-ca\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj\" (UID: \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:00.587758 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.587733    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e754ad-eb64-4930-9e55-abf3abaca1b3-openshift-service-ca\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj\" (UID: \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:00.589602 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.589578    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1e754ad-eb64-4930-9e55-abf3abaca1b3-metrics-certs\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj\" (UID: \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:00.614464 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:00.614438    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcmq6\" (UniqueName: \"kubernetes.io/projected/f1e754ad-eb64-4930-9e55-abf3abaca1b3-kube-api-access-xcmq6\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj\" (UID: \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:01.102793 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:01.102755    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1e754ad-eb64-4930-9e55-abf3abaca1b3-maas-api-tls\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj\" (UID: \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:01.105317 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:01.105292    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1e754ad-eb64-4930-9e55-abf3abaca1b3-maas-api-tls\") pod \"maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj\" (UID: \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:01.380207 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:01.380180    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d892r\""
Oct 30 19:48:01.386991 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:01.386967    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:01.654565 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:01.654495    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"]
Oct 30 19:48:01.659956 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:01.659938    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"
Oct 30 19:48:01.672931 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:01.672908    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2761c7\""
Oct 30 19:48:01.674518 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:01.674233    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"]
Oct 30 19:48:01.674518 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:01.674441    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2761c7-dockercfg-gghph\""
Oct 30 19:48:01.720639 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:01.720606    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6l85\" (UniqueName: \"kubernetes.io/projected/6927b283-075a-4da5-8fe1-3c019fed3c6b-kube-api-access-w6l85\") pod \"payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5\" (UID: \"6927b283-075a-4da5-8fe1-3c019fed3c6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"
Oct 30 19:48:01.720833 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:01.720672    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6927b283-075a-4da5-8fe1-3c019fed3c6b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5\" (UID: \"6927b283-075a-4da5-8fe1-3c019fed3c6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"
Oct 30 19:48:01.736079 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:01.733326    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"]
Oct 30 19:48:01.736079 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:48:01.734332    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e754ad_eb64_4930_9e55_abf3abaca1b3.slice/crio-7066ef8f5edcf8b03b592c772d722dc7323de7711ca453eb221f9cbfa836986f WatchSource:0}: Error finding container 7066ef8f5edcf8b03b592c772d722dc7323de7711ca453eb221f9cbfa836986f: Status 404 returned error can't find the container with id 7066ef8f5edcf8b03b592c772d722dc7323de7711ca453eb221f9cbfa836986f
Oct 30 19:48:01.828073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:01.823383    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6l85\" (UniqueName: \"kubernetes.io/projected/6927b283-075a-4da5-8fe1-3c019fed3c6b-kube-api-access-w6l85\") pod \"payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5\" (UID: \"6927b283-075a-4da5-8fe1-3c019fed3c6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"
Oct 30 19:48:01.828073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:01.823473    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6927b283-075a-4da5-8fe1-3c019fed3c6b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5\" (UID: \"6927b283-075a-4da5-8fe1-3c019fed3c6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"
Oct 30 19:48:01.828073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:01.824741    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6927b283-075a-4da5-8fe1-3c019fed3c6b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5\" (UID: \"6927b283-075a-4da5-8fe1-3c019fed3c6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"
Oct 30 19:48:01.867071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:01.866162    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6l85\" (UniqueName: \"kubernetes.io/projected/6927b283-075a-4da5-8fe1-3c019fed3c6b-kube-api-access-w6l85\") pod \"payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5\" (UID: \"6927b283-075a-4da5-8fe1-3c019fed3c6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"
Oct 30 19:48:01.984387 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:01.984305    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"
Oct 30 19:48:02.168927 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:02.168877    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj" event={"ID":"f1e754ad-eb64-4930-9e55-abf3abaca1b3","Type":"ContainerStarted","Data":"7066ef8f5edcf8b03b592c772d722dc7323de7711ca453eb221f9cbfa836986f"}
Oct 30 19:48:02.288166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:02.288139    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"]
Oct 30 19:48:02.289004 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:48:02.288973    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6927b283_075a_4da5_8fe1_3c019fed3c6b.slice/crio-6a9129049cd60bd6d168a688911783667ac18b03edeb5fff3e6184d6fc6ce669 WatchSource:0}: Error finding container 6a9129049cd60bd6d168a688911783667ac18b03edeb5fff3e6184d6fc6ce669: Status 404 returned error can't find the container with id 6a9129049cd60bd6d168a688911783667ac18b03edeb5fff3e6184d6fc6ce669
Oct 30 19:48:03.176843 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:03.176810    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5" event={"ID":"6927b283-075a-4da5-8fe1-3c019fed3c6b","Type":"ContainerStarted","Data":"0eeb30c6bace190ad349c9a0521003851678e9138119467736c22ab69a508829"}
Oct 30 19:48:03.177335 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:03.176886    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5" event={"ID":"6927b283-075a-4da5-8fe1-3c019fed3c6b","Type":"ContainerStarted","Data":"6a9129049cd60bd6d168a688911783667ac18b03edeb5fff3e6184d6fc6ce669"}
Oct 30 19:48:03.177335 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:03.176996    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"
Oct 30 19:48:03.178569 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:03.178544    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj" event={"ID":"f1e754ad-eb64-4930-9e55-abf3abaca1b3","Type":"ContainerStarted","Data":"e76aec2c6705e5f8dc127c0cf80764b0c4c66f5b41f30da4e1599032c276f163"}
Oct 30 19:48:03.178690 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:03.178597    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:03.212753 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:03.212687    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5" podStartSLOduration=2.21266999 podStartE2EDuration="2.21266999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:03.210767069 +0000 UTC m=+962.126373928" watchObservedRunningTime="2026-10-30 19:48:03.21266999 +0000 UTC m=+962.128276847"
Oct 30 19:48:03.256694 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:03.256643    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj" podStartSLOduration=2.755384746 podStartE2EDuration="3.256629746s" totalImagesPullingTime="501.245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:03.256190979 +0000 UTC m=+962.171797835" watchObservedRunningTime="2026-10-30 19:48:03.256629746 +0000 UTC m=+962.172236602"
Oct 30 19:48:04.068286 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:04.068255    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"
Oct 30 19:48:04.288451 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:48:04.288414    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:48:06.050909 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:06.050876    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"]
Oct 30 19:48:06.054824 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:06.054797    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5013423-65b7g"]
Oct 30 19:48:07.201706 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:07.201673    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57" path="/var/lib/kubelet/pods/8e1e7edb-32ea-4c0f-8b58-3b7668ae6c57/volumes"
Oct 30 19:48:08.664887 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:08.664854    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"]
Oct 30 19:48:08.670305 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:08.670280    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:48:08.681745 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:08.681696    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2761c7-serving-cert\""
Oct 30 19:48:08.684879 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:08.684850    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"]
Oct 30 19:48:08.814706 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:08.814673    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7a5e10-e02b-482b-81ac-21be2160fad9-maas-api-tls\") pod \"maas-api-e2e-api-b-2761c7-7999cf6469-psn25\" (UID: \"7d7a5e10-e02b-482b-81ac-21be2160fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:48:08.814905 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:08.814722    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7a5e10-e02b-482b-81ac-21be2160fad9-metrics-certs\") pod \"maas-api-e2e-api-b-2761c7-7999cf6469-psn25\" (UID: \"7d7a5e10-e02b-482b-81ac-21be2160fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:48:08.814905 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:08.814798    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g9vx\" (UniqueName: \"kubernetes.io/projected/7d7a5e10-e02b-482b-81ac-21be2160fad9-kube-api-access-6g9vx\") pod \"maas-api-e2e-api-b-2761c7-7999cf6469-psn25\" (UID: \"7d7a5e10-e02b-482b-81ac-21be2160fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:48:08.814905 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:08.814878    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7a5e10-e02b-482b-81ac-21be2160fad9-openshift-service-ca\") pod \"maas-api-e2e-api-b-2761c7-7999cf6469-psn25\" (UID: \"7d7a5e10-e02b-482b-81ac-21be2160fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:48:08.917795 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:08.917534    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7a5e10-e02b-482b-81ac-21be2160fad9-maas-api-tls\") pod \"maas-api-e2e-api-b-2761c7-7999cf6469-psn25\" (UID: \"7d7a5e10-e02b-482b-81ac-21be2160fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:48:08.918181 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:08.918152    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7a5e10-e02b-482b-81ac-21be2160fad9-metrics-certs\") pod \"maas-api-e2e-api-b-2761c7-7999cf6469-psn25\" (UID: \"7d7a5e10-e02b-482b-81ac-21be2160fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:48:08.918368 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:08.918350    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g9vx\" (UniqueName: \"kubernetes.io/projected/7d7a5e10-e02b-482b-81ac-21be2160fad9-kube-api-access-6g9vx\") pod \"maas-api-e2e-api-b-2761c7-7999cf6469-psn25\" (UID: \"7d7a5e10-e02b-482b-81ac-21be2160fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:48:08.918562 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:08.918544    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7a5e10-e02b-482b-81ac-21be2160fad9-openshift-service-ca\") pod \"maas-api-e2e-api-b-2761c7-7999cf6469-psn25\" (UID: \"7d7a5e10-e02b-482b-81ac-21be2160fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:48:08.919714 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:08.919692    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7a5e10-e02b-482b-81ac-21be2160fad9-openshift-service-ca\") pod \"maas-api-e2e-api-b-2761c7-7999cf6469-psn25\" (UID: \"7d7a5e10-e02b-482b-81ac-21be2160fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:48:08.927437 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:08.927401    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7a5e10-e02b-482b-81ac-21be2160fad9-maas-api-tls\") pod \"maas-api-e2e-api-b-2761c7-7999cf6469-psn25\" (UID: \"7d7a5e10-e02b-482b-81ac-21be2160fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:48:08.930251 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:08.930220    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7a5e10-e02b-482b-81ac-21be2160fad9-metrics-certs\") pod \"maas-api-e2e-api-b-2761c7-7999cf6469-psn25\" (UID: \"7d7a5e10-e02b-482b-81ac-21be2160fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:48:08.948580 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:08.948479    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g9vx\" (UniqueName: \"kubernetes.io/projected/7d7a5e10-e02b-482b-81ac-21be2160fad9-kube-api-access-6g9vx\") pod \"maas-api-e2e-api-b-2761c7-7999cf6469-psn25\" (UID: \"7d7a5e10-e02b-482b-81ac-21be2160fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:48:08.990428 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:08.990394    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:48:09.295397 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:09.294258    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:09.397693 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:09.397673    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"]
Oct 30 19:48:10.283134 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:10.283102    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25" event={"ID":"7d7a5e10-e02b-482b-81ac-21be2160fad9","Type":"ContainerStarted","Data":"d8b84bb19900d620bac3cb9faf0562c9b8f56960227e4acbd227cd2f70f99a8d"}
Oct 30 19:48:10.283134 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:10.283150    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25" event={"ID":"7d7a5e10-e02b-482b-81ac-21be2160fad9","Type":"ContainerStarted","Data":"ed970821000d50e8634abc9f39c01e1ecc348ef7a9575c3ac73e40eecdd0712c"}
Oct 30 19:48:10.283614 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:10.283232    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:48:10.306982 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:10.306912    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25" podStartSLOduration=1.919159523 podStartE2EDuration="2.306894s" totalImagesPullingTime="387.734477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:10.306706555 +0000 UTC m=+969.222313424" watchObservedRunningTime="2026-10-30 19:48:10.306894 +0000 UTC m=+969.222500857"
Oct 30 19:48:14.189345 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:14.189313    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"
Oct 30 19:48:16.307833 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:16.307802    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:48:32.529999 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:32.529847    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"]
Oct 30 19:48:34.289226 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:48:34.289185    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:48:37.544279 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:37.544238    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln" podUID="4fc9886a-2484-42d7-acde-a8724a17d8cb" containerName="payload-processing" containerID="cri-o://feb58bd4087a5dbb55261306dbe4e1e2942f87e491872872b1e50fdfd5c5571b" gracePeriod=25
Oct 30 19:48:37.830100 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:37.830065    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"
Oct 30 19:48:37.840615 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:37.840584    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fc9886a-2484-42d7-acde-a8724a17d8cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc9886a-2484-42d7-acde-a8724a17d8cb-plugins-config-volume\") pod \"4fc9886a-2484-42d7-acde-a8724a17d8cb\" (UID: \"4fc9886a-2484-42d7-acde-a8724a17d8cb\") "
Oct 30 19:48:37.840778 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:37.840731    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fc9886a-2484-42d7-acde-a8724a17d8cb-kube-api-access-94pxt\" (UniqueName: \"kubernetes.io/projected/4fc9886a-2484-42d7-acde-a8724a17d8cb-kube-api-access-94pxt\") pod \"4fc9886a-2484-42d7-acde-a8724a17d8cb\" (UID: \"4fc9886a-2484-42d7-acde-a8724a17d8cb\") "
Oct 30 19:48:37.841083 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:37.841012    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc9886a-2484-42d7-acde-a8724a17d8cb-plugins-config-volume" pod "4fc9886a-2484-42d7-acde-a8724a17d8cb" (UID: "4fc9886a-2484-42d7-acde-a8724a17d8cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:37.841385 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:37.841341    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc9886a-2484-42d7-acde-a8724a17d8cb-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:37.843103 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:37.843034    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc9886a-2484-42d7-acde-a8724a17d8cb-kube-api-access-94pxt" pod "4fc9886a-2484-42d7-acde-a8724a17d8cb" (UID: "4fc9886a-2484-42d7-acde-a8724a17d8cb"). InnerVolumeSpecName "kube-api-access-94pxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:37.942601 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:37.942567    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94pxt\" (UniqueName: \"kubernetes.io/projected/4fc9886a-2484-42d7-acde-a8724a17d8cb-kube-api-access-94pxt\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:38.735558 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:38.735516    6030 generic.go:356] "Generic (PLEG): container finished" podID="4fc9886a-2484-42d7-acde-a8724a17d8cb" containerID="feb58bd4087a5dbb55261306dbe4e1e2942f87e491872872b1e50fdfd5c5571b" exitCode=0
Oct 30 19:48:38.736336 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:38.735657    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"
Oct 30 19:48:38.745712 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:38.735699    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln" event={"ID":"4fc9886a-2484-42d7-acde-a8724a17d8cb","Type":"ContainerDied","Data":"feb58bd4087a5dbb55261306dbe4e1e2942f87e491872872b1e50fdfd5c5571b"}
Oct 30 19:48:38.745712 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:38.740887    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln" event={"ID":"4fc9886a-2484-42d7-acde-a8724a17d8cb","Type":"ContainerDied","Data":"a5d2b4c2d52bf16eb4e186354e8f95986a6dc723397582b310b026d668ac8f1b"}
Oct 30 19:48:38.745712 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:38.740929    6030 scope.go:122] "RemoveContainer" containerID="feb58bd4087a5dbb55261306dbe4e1e2942f87e491872872b1e50fdfd5c5571b"
Oct 30 19:48:38.767509 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:38.766807    6030 scope.go:122] "RemoveContainer" containerID="feb58bd4087a5dbb55261306dbe4e1e2942f87e491872872b1e50fdfd5c5571b"
Oct 30 19:48:38.767509 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:48:38.767212    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb58bd4087a5dbb55261306dbe4e1e2942f87e491872872b1e50fdfd5c5571b\": container with ID starting with feb58bd4087a5dbb55261306dbe4e1e2942f87e491872872b1e50fdfd5c5571b not found: ID does not exist" containerID="feb58bd4087a5dbb55261306dbe4e1e2942f87e491872872b1e50fdfd5c5571b"
Oct 30 19:48:38.767509 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:38.767251    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb58bd4087a5dbb55261306dbe4e1e2942f87e491872872b1e50fdfd5c5571b"} err="failed to get container status \"feb58bd4087a5dbb55261306dbe4e1e2942f87e491872872b1e50fdfd5c5571b\": rpc error: code = NotFound desc = could not find container \"feb58bd4087a5dbb55261306dbe4e1e2942f87e491872872b1e50fdfd5c5571b\": container with ID starting with feb58bd4087a5dbb55261306dbe4e1e2942f87e491872872b1e50fdfd5c5571b not found: ID does not exist"
Oct 30 19:48:38.799102 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:38.799076    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"]
Oct 30 19:48:38.809272 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:38.809249    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-477fd9c0-85c844f66d-x6xln"]
Oct 30 19:48:39.202744 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:39.202702    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fc9886a-2484-42d7-acde-a8724a17d8cb" path="/var/lib/kubelet/pods/4fc9886a-2484-42d7-acde-a8724a17d8cb/volumes"
Oct 30 19:48:40.617157 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:40.617125    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"]
Oct 30 19:48:40.617607 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:40.617340    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj" podUID="f1e754ad-eb64-4930-9e55-abf3abaca1b3" containerName="maas-api" containerID="cri-o://e76aec2c6705e5f8dc127c0cf80764b0c4c66f5b41f30da4e1599032c276f163" gracePeriod=30
Oct 30 19:48:40.756873 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:40.755556    6030 generic.go:356] "Generic (PLEG): container finished" podID="f1e754ad-eb64-4930-9e55-abf3abaca1b3" containerID="e76aec2c6705e5f8dc127c0cf80764b0c4c66f5b41f30da4e1599032c276f163" exitCode=0
Oct 30 19:48:40.757063 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:40.756971    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj" event={"ID":"f1e754ad-eb64-4930-9e55-abf3abaca1b3","Type":"ContainerDied","Data":"e76aec2c6705e5f8dc127c0cf80764b0c4c66f5b41f30da4e1599032c276f163"}
Oct 30 19:48:41.114017 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.113985    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:41.243831 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.243801    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1e754ad-eb64-4930-9e55-abf3abaca1b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e754ad-eb64-4930-9e55-abf3abaca1b3-openshift-service-ca\") pod \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\" (UID: \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\") "
Oct 30 19:48:41.243988 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.243882    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1e754ad-eb64-4930-9e55-abf3abaca1b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1e754ad-eb64-4930-9e55-abf3abaca1b3-metrics-certs\") pod \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\" (UID: \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\") "
Oct 30 19:48:41.243988 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.243950    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1e754ad-eb64-4930-9e55-abf3abaca1b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1e754ad-eb64-4930-9e55-abf3abaca1b3-maas-api-tls\") pod \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\" (UID: \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\") "
Oct 30 19:48:41.244127 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.244009    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e754ad-eb64-4930-9e55-abf3abaca1b3-kube-api-access-xcmq6\" (UniqueName: \"kubernetes.io/projected/f1e754ad-eb64-4930-9e55-abf3abaca1b3-kube-api-access-xcmq6\") pod \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\" (UID: \"f1e754ad-eb64-4930-9e55-abf3abaca1b3\") "
Oct 30 19:48:41.244195 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.244136    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e754ad-eb64-4930-9e55-abf3abaca1b3-openshift-service-ca" pod "f1e754ad-eb64-4930-9e55-abf3abaca1b3" (UID: "f1e754ad-eb64-4930-9e55-abf3abaca1b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:41.244600 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.244527    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e754ad-eb64-4930-9e55-abf3abaca1b3-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:41.246436 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.246408    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e754ad-eb64-4930-9e55-abf3abaca1b3-kube-api-access-xcmq6" pod "f1e754ad-eb64-4930-9e55-abf3abaca1b3" (UID: "f1e754ad-eb64-4930-9e55-abf3abaca1b3"). InnerVolumeSpecName "kube-api-access-xcmq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:41.246585 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.246570    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1e754ad-eb64-4930-9e55-abf3abaca1b3-maas-api-tls" pod "f1e754ad-eb64-4930-9e55-abf3abaca1b3" (UID: "f1e754ad-eb64-4930-9e55-abf3abaca1b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:41.246633 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.246591    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1e754ad-eb64-4930-9e55-abf3abaca1b3-metrics-certs" pod "f1e754ad-eb64-4930-9e55-abf3abaca1b3" (UID: "f1e754ad-eb64-4930-9e55-abf3abaca1b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:41.349196 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.348071    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1e754ad-eb64-4930-9e55-abf3abaca1b3-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:41.349196 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.348106    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1e754ad-eb64-4930-9e55-abf3abaca1b3-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:41.349196 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.348123    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcmq6\" (UniqueName: \"kubernetes.io/projected/f1e754ad-eb64-4930-9e55-abf3abaca1b3-kube-api-access-xcmq6\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:41.764269 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.764238    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj" event={"ID":"f1e754ad-eb64-4930-9e55-abf3abaca1b3","Type":"ContainerDied","Data":"7066ef8f5edcf8b03b592c772d722dc7323de7711ca453eb221f9cbfa836986f"}
Oct 30 19:48:41.764723 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.764299    6030 scope.go:122] "RemoveContainer" containerID="e76aec2c6705e5f8dc127c0cf80764b0c4c66f5b41f30da4e1599032c276f163"
Oct 30 19:48:41.764723 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.764250    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"
Oct 30 19:48:41.800918 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.800887    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"]
Oct 30 19:48:41.808181 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:41.808153    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-477fd9c0-5f78d65fff-m2qkj"]
Oct 30 19:48:43.202200 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:43.202172    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e754ad-eb64-4930-9e55-abf3abaca1b3" path="/var/lib/kubelet/pods/f1e754ad-eb64-4930-9e55-abf3abaca1b3/volumes"
Oct 30 19:48:46.654755 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:46.648726    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"]
Oct 30 19:48:47.126120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.108410    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"]
Oct 30 19:48:47.126120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.109212    6030 cpu_manager.go:414] "containerMap: removing container" podUID="4fc9886a-2484-42d7-acde-a8724a17d8cb" containerName="payload-processing"
Oct 30 19:48:47.126120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.109232    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fc9886a-2484-42d7-acde-a8724a17d8cb" containerName="payload-processing"
Oct 30 19:48:47.126120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.109263    6030 cpu_manager.go:414] "containerMap: removing container" podUID="f1e754ad-eb64-4930-9e55-abf3abaca1b3" containerName="maas-api"
Oct 30 19:48:47.126120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.109272    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e754ad-eb64-4930-9e55-abf3abaca1b3" containerName="maas-api"
Oct 30 19:48:47.126120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.109399    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e754ad-eb64-4930-9e55-abf3abaca1b3" containerName="maas-api"
Oct 30 19:48:47.126120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.109420    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fc9886a-2484-42d7-acde-a8724a17d8cb" containerName="payload-processing"
Oct 30 19:48:47.126120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.119385    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.132522 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.130762    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c9d69c-openshift-default-dockercfg-clfxv\""
Oct 30 19:48:47.137818 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.137788    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"]
Oct 30 19:48:47.229072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.228787    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-token\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.229072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.228835    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istiod-ca-cert\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.229072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.228933    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-podinfo\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.229072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.228972    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-envoy\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.229072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.229009    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58zwg\" (UniqueName: \"kubernetes.io/projected/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-kube-api-access-58zwg\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.229072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.229078    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-credential-socket\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.229552 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.229158    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-workload-socket\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.229552 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.229214    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-data\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.229552 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.229292    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-workload-certs\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.333068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.332949    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-token\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.333068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.333013    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istiod-ca-cert\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.333335 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.333147    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-podinfo\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.333335 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.333198    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-envoy\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.333335 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.333238    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58zwg\" (UniqueName: \"kubernetes.io/projected/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-kube-api-access-58zwg\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.333335 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.333288    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-credential-socket\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.333607 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.333369    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-workload-socket\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.333607 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.333433    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-data\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.333607 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.333513    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-workload-certs\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.358084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.333958    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-workload-certs\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.358084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.335076    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istiod-ca-cert\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.358084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.335716    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-credential-socket\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.358084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.336387    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-workload-socket\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.358084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.336893    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-data\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.370068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.369738    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-token\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.396510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.386379    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-envoy\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.396510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.395012    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-podinfo\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.400652 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.400625    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58zwg\" (UniqueName: \"kubernetes.io/projected/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-kube-api-access-58zwg\") pod \"e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") " pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:47.454336 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:47.454298    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:48.086791 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:48.081920    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"]
Oct 30 19:48:48.112252 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:48:48.112213    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45eef3eb_e56b_4e28_ae5c_02b5eeae51b8.slice/crio-761d1a89c1a25035894b25f30a7a0b78f8b2374d384c200006f4789ab84747ce WatchSource:0}: Error finding container 761d1a89c1a25035894b25f30a7a0b78f8b2374d384c200006f4789ab84747ce: Status 404 returned error can't find the container with id 761d1a89c1a25035894b25f30a7a0b78f8b2374d384c200006f4789ab84747ce
Oct 30 19:48:48.115401 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:48.115359    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:48:48.115518 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:48.115439    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:48:48.115518 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:48.115478    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:48:48.869342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:48.869300    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx" event={"ID":"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8","Type":"ContainerStarted","Data":"2d85ef91684daaf90ad19f582411a2b74d4ff3f815567cabc3f0a3d0877d5a5b"}
Oct 30 19:48:48.869581 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:48.869400    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx" event={"ID":"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8","Type":"ContainerStarted","Data":"761d1a89c1a25035894b25f30a7a0b78f8b2374d384c200006f4789ab84747ce"}
Oct 30 19:48:48.912175 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:48.912122    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx" podStartSLOduration=2.912108837 podStartE2EDuration="2.912108837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:48.910752539 +0000 UTC m=+1007.826359393" watchObservedRunningTime="2026-10-30 19:48:48.912108837 +0000 UTC m=+1007.827715692"
Oct 30 19:48:49.460071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:49.459227    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:49.482595 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:49.480653    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:49.875749 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:49.875722    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:49.876661 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:49.876641    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:48:58.800072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:58.797103    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"]
Oct 30 19:48:58.882572 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:58.882540    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"
Oct 30 19:48:58.884356 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:58.883959    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"]
Oct 30 19:48:58.893185 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:58.893157    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-913351\""
Oct 30 19:48:58.893399 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:58.893377    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-913351-dockercfg-kgqrt\""
Oct 30 19:48:58.955400 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:58.955363    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0204dc-bca5-497c-8d32-1057e9e0af46-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s\" (UID: \"2a0204dc-bca5-497c-8d32-1057e9e0af46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"
Oct 30 19:48:58.955400 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:58.955409    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsqwh\" (UniqueName: \"kubernetes.io/projected/2a0204dc-bca5-497c-8d32-1057e9e0af46-kube-api-access-rsqwh\") pod \"payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s\" (UID: \"2a0204dc-bca5-497c-8d32-1057e9e0af46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"
Oct 30 19:48:59.056322 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:59.056234    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0204dc-bca5-497c-8d32-1057e9e0af46-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s\" (UID: \"2a0204dc-bca5-497c-8d32-1057e9e0af46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"
Oct 30 19:48:59.056322 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:59.056280    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsqwh\" (UniqueName: \"kubernetes.io/projected/2a0204dc-bca5-497c-8d32-1057e9e0af46-kube-api-access-rsqwh\") pod \"payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s\" (UID: \"2a0204dc-bca5-497c-8d32-1057e9e0af46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"
Oct 30 19:48:59.056993 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:59.056973    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0204dc-bca5-497c-8d32-1057e9e0af46-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s\" (UID: \"2a0204dc-bca5-497c-8d32-1057e9e0af46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"
Oct 30 19:48:59.072085 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:59.072031    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsqwh\" (UniqueName: \"kubernetes.io/projected/2a0204dc-bca5-497c-8d32-1057e9e0af46-kube-api-access-rsqwh\") pod \"payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s\" (UID: \"2a0204dc-bca5-497c-8d32-1057e9e0af46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"
Oct 30 19:48:59.208448 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:59.208414    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"
Oct 30 19:48:59.577814 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:48:59.556859    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"]
Oct 30 19:48:59.581188 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:48:59.581145    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0204dc_bca5_497c_8d32_1057e9e0af46.slice/crio-e3ea015f95d8bbbf8ed38d7f4dae42745bdd93956e82a5b31aad0da8b3dbebf7 WatchSource:0}: Error finding container e3ea015f95d8bbbf8ed38d7f4dae42745bdd93956e82a5b31aad0da8b3dbebf7: Status 404 returned error can't find the container with id e3ea015f95d8bbbf8ed38d7f4dae42745bdd93956e82a5b31aad0da8b3dbebf7
Oct 30 19:49:00.037652 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:00.037614    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s" event={"ID":"2a0204dc-bca5-497c-8d32-1057e9e0af46","Type":"ContainerStarted","Data":"e3ea015f95d8bbbf8ed38d7f4dae42745bdd93956e82a5b31aad0da8b3dbebf7"}
Oct 30 19:49:01.447800 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:01.447756    6030 scope.go:122] "RemoveContainer" containerID="1613cf0816eecbae0f44da51d8ec11ecc144dcce031d474aa8e291e27f577716"
Oct 30 19:49:01.460915 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:01.460883    6030 scope.go:122] "RemoveContainer" containerID="2c3ce081b4d10e685e209b07a6992cbd1f07f189dbdc6240396c31f8b23de433"
Oct 30 19:49:01.475682 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:01.475656    6030 scope.go:122] "RemoveContainer" containerID="aea62fc534e99a7284ba7235e84c9c0c48c96928d0f33c5bb5a960ff8d7def9e"
Oct 30 19:49:01.580434 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:01.580267    6030 scope.go:122] "RemoveContainer" containerID="3ce89d4d2f60479326ea1031cdc5512f5ad7d05042a0777da497673f0fbbd698"
Oct 30 19:49:01.671270 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:01.668196    6030 scope.go:122] "RemoveContainer" containerID="cc3658058ee347e1cbe956df9e06f5ce45b759a12bae92f3967e16335fe3c708"
Oct 30 19:49:01.699732 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:01.699710    6030 scope.go:122] "RemoveContainer" containerID="436a508b3c22cfafc879bfb49b6c60d4e82e7c43d1bc127fb40d1ffea6012b7c"
Oct 30 19:49:01.749269 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:01.749198    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm" podUID="34dbd6be-7d09-4527-b23d-d3bc899a510d" containerName="main" containerID="cri-o://2359d25006fac502a0e379052c56881547cd013b3a38935460b75b9f70af55c0" gracePeriod=45
Oct 30 19:49:02.007180 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:02.006513    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm" podUID="34dbd6be-7d09-4527-b23d-d3bc899a510d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": dial tcp 10.133.0.79:8000: connect: connection refused"
Oct 30 19:49:02.222071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:02.221271    6030 generic.go:356] "Generic (PLEG): container finished" podID="34dbd6be-7d09-4527-b23d-d3bc899a510d" containerID="2359d25006fac502a0e379052c56881547cd013b3a38935460b75b9f70af55c0" exitCode=0
Oct 30 19:49:02.222071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:02.221578    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm" event={"ID":"34dbd6be-7d09-4527-b23d-d3bc899a510d","Type":"ContainerDied","Data":"2359d25006fac502a0e379052c56881547cd013b3a38935460b75b9f70af55c0"}
Oct 30 19:49:02.762201 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:02.761868    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:49:02.898341 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:02.898260    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-dshm\") pod \"34dbd6be-7d09-4527-b23d-d3bc899a510d\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") "
Oct 30 19:49:02.898341 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:02.898333    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-tmp-dir\") pod \"34dbd6be-7d09-4527-b23d-d3bc899a510d\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") "
Oct 30 19:49:02.898584 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:02.898392    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34dbd6be-7d09-4527-b23d-d3bc899a510d-kube-api-access-fs9wj\" (UniqueName: \"kubernetes.io/projected/34dbd6be-7d09-4527-b23d-d3bc899a510d-kube-api-access-fs9wj\") pod \"34dbd6be-7d09-4527-b23d-d3bc899a510d\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") "
Oct 30 19:49:02.898584 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:02.898416    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34dbd6be-7d09-4527-b23d-d3bc899a510d-tls-certs\" (UniqueName: \"kubernetes.io/secret/34dbd6be-7d09-4527-b23d-d3bc899a510d-tls-certs\") pod \"34dbd6be-7d09-4527-b23d-d3bc899a510d\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") "
Oct 30 19:49:02.898584 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:02.898524    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-model-cache\") pod \"34dbd6be-7d09-4527-b23d-d3bc899a510d\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") "
Oct 30 19:49:02.898748 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:02.898661    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-home\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-home\") pod \"34dbd6be-7d09-4527-b23d-d3bc899a510d\" (UID: \"34dbd6be-7d09-4527-b23d-d3bc899a510d\") "
Oct 30 19:49:02.921166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:02.900106    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-model-cache" pod "34dbd6be-7d09-4527-b23d-d3bc899a510d" (UID: "34dbd6be-7d09-4527-b23d-d3bc899a510d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:02.921166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:02.900358    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-tmp-dir" pod "34dbd6be-7d09-4527-b23d-d3bc899a510d" (UID: "34dbd6be-7d09-4527-b23d-d3bc899a510d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:02.946274 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:02.946155    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-home" pod "34dbd6be-7d09-4527-b23d-d3bc899a510d" (UID: "34dbd6be-7d09-4527-b23d-d3bc899a510d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:02.997699 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:02.997609    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34dbd6be-7d09-4527-b23d-d3bc899a510d-tls-certs" pod "34dbd6be-7d09-4527-b23d-d3bc899a510d" (UID: "34dbd6be-7d09-4527-b23d-d3bc899a510d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:03.002463 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:03.002224    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34dbd6be-7d09-4527-b23d-d3bc899a510d-kube-api-access-fs9wj" pod "34dbd6be-7d09-4527-b23d-d3bc899a510d" (UID: "34dbd6be-7d09-4527-b23d-d3bc899a510d"). InnerVolumeSpecName "kube-api-access-fs9wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:03.002600 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:03.002582    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:03.002663 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:03.002608    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs9wj\" (UniqueName: \"kubernetes.io/projected/34dbd6be-7d09-4527-b23d-d3bc899a510d-kube-api-access-fs9wj\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:03.002663 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:03.002624    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34dbd6be-7d09-4527-b23d-d3bc899a510d-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:03.002663 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:03.002638    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:03.002663 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:03.002652    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:03.026192 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:03.026136    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-dshm" pod "34dbd6be-7d09-4527-b23d-d3bc899a510d" (UID: "34dbd6be-7d09-4527-b23d-d3bc899a510d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:03.124084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:03.107079    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34dbd6be-7d09-4527-b23d-d3bc899a510d-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:03.261341 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:03.261117    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm" event={"ID":"34dbd6be-7d09-4527-b23d-d3bc899a510d","Type":"ContainerDied","Data":"0bb34b58414279eed862db66b8a7241a96f5532e01baf3b16fc8e7d1249aafdc"}
Oct 30 19:49:03.261341 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:03.261175    6030 scope.go:122] "RemoveContainer" containerID="2359d25006fac502a0e379052c56881547cd013b3a38935460b75b9f70af55c0"
Oct 30 19:49:03.261564 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:03.261345    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"
Oct 30 19:49:03.328150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:03.328074    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"]
Oct 30 19:49:03.358071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:03.357774    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-477fd9c0/e2e-lifecycle-model-477fd9c0-kserve-774fb7697c-xq4pm"]
Oct 30 19:49:04.272080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:04.271681    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s" event={"ID":"2a0204dc-bca5-497c-8d32-1057e9e0af46","Type":"ContainerStarted","Data":"99a0022360aebe6aa27184ce417944fb549dce392057563aba734eaed55a450c"}
Oct 30 19:49:04.272080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:04.271868    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"
Oct 30 19:49:04.291764 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:49:04.291727    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:49:04.295010 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:04.294967    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s" podStartSLOduration=2.4925549670000002 podStartE2EDuration="6.294949269s" totalImagesPullingTime="3.802394302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:04.292822176 +0000 UTC m=+1023.208429033" watchObservedRunningTime="2026-10-30 19:49:04.294949269 +0000 UTC m=+1023.210556176"
Oct 30 19:49:05.200945 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:05.200914    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34dbd6be-7d09-4527-b23d-d3bc899a510d" path="/var/lib/kubelet/pods/34dbd6be-7d09-4527-b23d-d3bc899a510d/volumes"
Oct 30 19:49:07.535703 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:07.535669    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"]
Oct 30 19:49:12.536908 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:12.536846    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5" podUID="6927b283-075a-4da5-8fe1-3c019fed3c6b" containerName="payload-processing" containerID="cri-o://0eeb30c6bace190ad349c9a0521003851678e9138119467736c22ab69a508829" gracePeriod=25
Oct 30 19:49:12.804371 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:12.804311    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"
Oct 30 19:49:12.920351 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:12.920315    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6927b283-075a-4da5-8fe1-3c019fed3c6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6927b283-075a-4da5-8fe1-3c019fed3c6b-plugins-config-volume\") pod \"6927b283-075a-4da5-8fe1-3c019fed3c6b\" (UID: \"6927b283-075a-4da5-8fe1-3c019fed3c6b\") "
Oct 30 19:49:12.920525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:12.920492    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6927b283-075a-4da5-8fe1-3c019fed3c6b-kube-api-access-w6l85\" (UniqueName: \"kubernetes.io/projected/6927b283-075a-4da5-8fe1-3c019fed3c6b-kube-api-access-w6l85\") pod \"6927b283-075a-4da5-8fe1-3c019fed3c6b\" (UID: \"6927b283-075a-4da5-8fe1-3c019fed3c6b\") "
Oct 30 19:49:12.920717 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:12.920687    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6927b283-075a-4da5-8fe1-3c019fed3c6b-plugins-config-volume" pod "6927b283-075a-4da5-8fe1-3c019fed3c6b" (UID: "6927b283-075a-4da5-8fe1-3c019fed3c6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:12.920999 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:12.920884    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6927b283-075a-4da5-8fe1-3c019fed3c6b-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:12.922999 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:12.922954    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6927b283-075a-4da5-8fe1-3c019fed3c6b-kube-api-access-w6l85" pod "6927b283-075a-4da5-8fe1-3c019fed3c6b" (UID: "6927b283-075a-4da5-8fe1-3c019fed3c6b"). InnerVolumeSpecName "kube-api-access-w6l85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:13.022569 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:13.022533    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6l85\" (UniqueName: \"kubernetes.io/projected/6927b283-075a-4da5-8fe1-3c019fed3c6b-kube-api-access-w6l85\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:13.337612 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:13.337573    6030 generic.go:356] "Generic (PLEG): container finished" podID="6927b283-075a-4da5-8fe1-3c019fed3c6b" containerID="0eeb30c6bace190ad349c9a0521003851678e9138119467736c22ab69a508829" exitCode=0
Oct 30 19:49:13.337817 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:13.337625    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"
Oct 30 19:49:13.337817 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:13.337641    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5" event={"ID":"6927b283-075a-4da5-8fe1-3c019fed3c6b","Type":"ContainerDied","Data":"0eeb30c6bace190ad349c9a0521003851678e9138119467736c22ab69a508829"}
Oct 30 19:49:13.337817 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:13.337686    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5" event={"ID":"6927b283-075a-4da5-8fe1-3c019fed3c6b","Type":"ContainerDied","Data":"6a9129049cd60bd6d168a688911783667ac18b03edeb5fff3e6184d6fc6ce669"}
Oct 30 19:49:13.337817 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:13.337708    6030 scope.go:122] "RemoveContainer" containerID="0eeb30c6bace190ad349c9a0521003851678e9138119467736c22ab69a508829"
Oct 30 19:49:13.349731 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:13.349714    6030 scope.go:122] "RemoveContainer" containerID="0eeb30c6bace190ad349c9a0521003851678e9138119467736c22ab69a508829"
Oct 30 19:49:13.350090 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:49:13.350034    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eeb30c6bace190ad349c9a0521003851678e9138119467736c22ab69a508829\": container with ID starting with 0eeb30c6bace190ad349c9a0521003851678e9138119467736c22ab69a508829 not found: ID does not exist" containerID="0eeb30c6bace190ad349c9a0521003851678e9138119467736c22ab69a508829"
Oct 30 19:49:13.350187 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:13.350098    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eeb30c6bace190ad349c9a0521003851678e9138119467736c22ab69a508829"} err="failed to get container status \"0eeb30c6bace190ad349c9a0521003851678e9138119467736c22ab69a508829\": rpc error: code = NotFound desc = could not find container \"0eeb30c6bace190ad349c9a0521003851678e9138119467736c22ab69a508829\": container with ID starting with 0eeb30c6bace190ad349c9a0521003851678e9138119467736c22ab69a508829 not found: ID does not exist"
Oct 30 19:49:13.362005 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:13.361981    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"]
Oct 30 19:49:13.367060 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:13.367027    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2761c7-86c888b596-7b8d5"]
Oct 30 19:49:15.201486 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:15.201448    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6927b283-075a-4da5-8fe1-3c019fed3c6b" path="/var/lib/kubelet/pods/6927b283-075a-4da5-8fe1-3c019fed3c6b/volumes"
Oct 30 19:49:15.281221 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:15.281190    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"
Oct 30 19:49:17.687156 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.687117    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"]
Oct 30 19:49:17.687871 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.687837    6030 cpu_manager.go:414] "containerMap: removing container" podUID="6927b283-075a-4da5-8fe1-3c019fed3c6b" containerName="payload-processing"
Oct 30 19:49:17.687871 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.687860    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6927b283-075a-4da5-8fe1-3c019fed3c6b" containerName="payload-processing"
Oct 30 19:49:17.687871 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.687871    6030 cpu_manager.go:414] "containerMap: removing container" podUID="34dbd6be-7d09-4527-b23d-d3bc899a510d" containerName="main"
Oct 30 19:49:17.687871 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.687879    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34dbd6be-7d09-4527-b23d-d3bc899a510d" containerName="main"
Oct 30 19:49:17.688217 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.688018    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="6927b283-075a-4da5-8fe1-3c019fed3c6b" containerName="payload-processing"
Oct 30 19:49:17.688217 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.688029    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="34dbd6be-7d09-4527-b23d-d3bc899a510d" containerName="main"
Oct 30 19:49:17.693881 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.693855    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.709016 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.708988    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-59e21a-openshift-default-dockercfg-f5kpx\""
Oct 30 19:49:17.713956 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.713930    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"]
Oct 30 19:49:17.811391 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.811354    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-data\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.811566 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.811502    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-credential-socket\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.811607 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.811586    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-podinfo\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.811642 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.811625    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-token\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.811676 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.811654    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-envoy\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.811787 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.811766    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-workload-socket\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.811825 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.811798    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdd5p\" (UniqueName: \"kubernetes.io/projected/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-kube-api-access-cdd5p\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.811858 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.811823    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-workload-certs\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.811987 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.811956    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istiod-ca-cert\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.924070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.916902    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istiod-ca-cert\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.924070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.916978    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-data\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.924070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.917130    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-credential-socket\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.924070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.917212    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-podinfo\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.924070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.917255    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-token\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.924070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.917289    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-envoy\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.924070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.917332    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-workload-socket\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.924070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.917359    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdd5p\" (UniqueName: \"kubernetes.io/projected/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-kube-api-access-cdd5p\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.924070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.917392    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-workload-certs\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.924070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.917863    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-workload-certs\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.924070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.918539    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istiod-ca-cert\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.924070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.918784    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-data\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.924070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.919020    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-credential-socket\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.924070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.919903    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-workload-socket\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.935403 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.935368    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-envoy\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.982175 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.979669    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-podinfo\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:17.990682 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.990644    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-token\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:18.006069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:17.991722    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdd5p\" (UniqueName: \"kubernetes.io/projected/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-kube-api-access-cdd5p\") pod \"e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") " pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:18.016549 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:18.016285    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:18.378478 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:18.378450    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"]
Oct 30 19:49:18.379300 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:49:18.379265    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de060ca_04d8_4c9e_aff5_78d3dbbc0199.slice/crio-40fdce100fb16bee61d8e4a8156d37958e4a05d0029e9a726768cba25b18d56c WatchSource:0}: Error finding container 40fdce100fb16bee61d8e4a8156d37958e4a05d0029e9a726768cba25b18d56c: Status 404 returned error can't find the container with id 40fdce100fb16bee61d8e4a8156d37958e4a05d0029e9a726768cba25b18d56c
Oct 30 19:49:18.381989 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:18.381950    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:49:18.382101 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:18.382066    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:49:18.382166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:18.382117    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:49:19.379414 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.379380    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj" event={"ID":"8de060ca-04d8-4c9e-aff5-78d3dbbc0199","Type":"ContainerStarted","Data":"c580b2b203e5f0e9ac68df6aade37d1facda218fd802604e2ea2593f2732c2bc"}
Oct 30 19:49:19.379811 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.379429    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj" event={"ID":"8de060ca-04d8-4c9e-aff5-78d3dbbc0199","Type":"ContainerStarted","Data":"40fdce100fb16bee61d8e4a8156d37958e4a05d0029e9a726768cba25b18d56c"}
Oct 30 19:49:19.437926 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.437866    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj" podStartSLOduration=2.437845621 podStartE2EDuration="2.437845621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:19.43093166 +0000 UTC m=+1038.346538509" watchObservedRunningTime="2026-10-30 19:49:19.437845621 +0000 UTC m=+1038.353452477"
Oct 30 19:49:19.541352 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.541319    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"]
Oct 30 19:49:19.545506 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.545483    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"
Oct 30 19:49:19.551204 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.551183    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1e3723-dockercfg-ldsnc\""
Oct 30 19:49:19.551343 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.551326    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1e3723\""
Oct 30 19:49:19.558273 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.558249    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"]
Oct 30 19:49:19.584865 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.584834    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"]
Oct 30 19:49:19.588795 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.588773    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"
Oct 30 19:49:19.601184 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.601161    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"]
Oct 30 19:49:19.661119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.657181    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6f202f-9bf1-4e5e-b399-bbaf83e2adce-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc\" (UID: \"1f6f202f-9bf1-4e5e-b399-bbaf83e2adce\") " pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"
Oct 30 19:49:19.661119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.657373    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj5p8\" (UniqueName: \"kubernetes.io/projected/1f6f202f-9bf1-4e5e-b399-bbaf83e2adce-kube-api-access-zj5p8\") pod \"payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc\" (UID: \"1f6f202f-9bf1-4e5e-b399-bbaf83e2adce\") " pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"
Oct 30 19:49:19.781121 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.765842    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj5p8\" (UniqueName: \"kubernetes.io/projected/1f6f202f-9bf1-4e5e-b399-bbaf83e2adce-kube-api-access-zj5p8\") pod \"payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc\" (UID: \"1f6f202f-9bf1-4e5e-b399-bbaf83e2adce\") " pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"
Oct 30 19:49:19.802125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.801195    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzsqr\" (UniqueName: \"kubernetes.io/projected/a70f60f8-d58b-4db3-a200-4958351a5e47-kube-api-access-kzsqr\") pod \"payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx\" (UID: \"a70f60f8-d58b-4db3-a200-4958351a5e47\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"
Oct 30 19:49:19.802125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.801396    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70f60f8-d58b-4db3-a200-4958351a5e47-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx\" (UID: \"a70f60f8-d58b-4db3-a200-4958351a5e47\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"
Oct 30 19:49:19.802125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.801451    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6f202f-9bf1-4e5e-b399-bbaf83e2adce-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc\" (UID: \"1f6f202f-9bf1-4e5e-b399-bbaf83e2adce\") " pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"
Oct 30 19:49:19.818113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.802537    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6f202f-9bf1-4e5e-b399-bbaf83e2adce-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc\" (UID: \"1f6f202f-9bf1-4e5e-b399-bbaf83e2adce\") " pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"
Oct 30 19:49:19.847546 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.843818    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj5p8\" (UniqueName: \"kubernetes.io/projected/1f6f202f-9bf1-4e5e-b399-bbaf83e2adce-kube-api-access-zj5p8\") pod \"payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc\" (UID: \"1f6f202f-9bf1-4e5e-b399-bbaf83e2adce\") " pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"
Oct 30 19:49:19.871067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.863590    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"
Oct 30 19:49:19.904712 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.902526    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70f60f8-d58b-4db3-a200-4958351a5e47-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx\" (UID: \"a70f60f8-d58b-4db3-a200-4958351a5e47\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"
Oct 30 19:49:19.904712 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.902756    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzsqr\" (UniqueName: \"kubernetes.io/projected/a70f60f8-d58b-4db3-a200-4958351a5e47-kube-api-access-kzsqr\") pod \"payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx\" (UID: \"a70f60f8-d58b-4db3-a200-4958351a5e47\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"
Oct 30 19:49:19.904712 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.904014    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70f60f8-d58b-4db3-a200-4958351a5e47-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx\" (UID: \"a70f60f8-d58b-4db3-a200-4958351a5e47\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"
Oct 30 19:49:19.929402 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:19.929360    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzsqr\" (UniqueName: \"kubernetes.io/projected/a70f60f8-d58b-4db3-a200-4958351a5e47-kube-api-access-kzsqr\") pod \"payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx\" (UID: \"a70f60f8-d58b-4db3-a200-4958351a5e47\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"
Oct 30 19:49:20.017404 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:20.017363    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:20.024341 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:20.024313    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:20.216132 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:20.211253    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"
Oct 30 19:49:20.252442 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:20.252411    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"]
Oct 30 19:49:20.252587 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:49:20.252538    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6f202f_9bf1_4e5e_b399_bbaf83e2adce.slice/crio-964e2a78cad4420c6bb1b402c69691034bb837cbff2dfb63f3625e488fe82b4b WatchSource:0}: Error finding container 964e2a78cad4420c6bb1b402c69691034bb837cbff2dfb63f3625e488fe82b4b: Status 404 returned error can't find the container with id 964e2a78cad4420c6bb1b402c69691034bb837cbff2dfb63f3625e488fe82b4b
Oct 30 19:49:20.386999 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:20.386955    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc" event={"ID":"1f6f202f-9bf1-4e5e-b399-bbaf83e2adce","Type":"ContainerStarted","Data":"5b931bddbffce81255c121c96737bb3f8d108f9f7f4837dfe3c933c5dbe4de77"}
Oct 30 19:49:20.387494 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:20.387011    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc" event={"ID":"1f6f202f-9bf1-4e5e-b399-bbaf83e2adce","Type":"ContainerStarted","Data":"964e2a78cad4420c6bb1b402c69691034bb837cbff2dfb63f3625e488fe82b4b"}
Oct 30 19:49:20.387494 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:20.387266    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"
Oct 30 19:49:20.387494 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:20.387300    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:20.388640 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:20.388620    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:49:20.409696 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:20.409645    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc" podStartSLOduration=1.4096318380000001 podStartE2EDuration="1.409631838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:20.407559694 +0000 UTC m=+1039.323166575" watchObservedRunningTime="2026-10-30 19:49:20.409631838 +0000 UTC m=+1039.325238693"
Oct 30 19:49:20.515933 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:20.515902    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"]
Oct 30 19:49:20.516659 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:49:20.516635    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70f60f8_d58b_4db3_a200_4958351a5e47.slice/crio-8befd2a16a12e7d868ab9d93b08878d914553b0ce363c3c0888203ba9c6c20da WatchSource:0}: Error finding container 8befd2a16a12e7d868ab9d93b08878d914553b0ce363c3c0888203ba9c6c20da: Status 404 returned error can't find the container with id 8befd2a16a12e7d868ab9d93b08878d914553b0ce363c3c0888203ba9c6c20da
Oct 30 19:49:21.395910 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:21.395870    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx" event={"ID":"a70f60f8-d58b-4db3-a200-4958351a5e47","Type":"ContainerStarted","Data":"d53f53c1bac89f524de63b59fa20c7029004113bfd34f61729c7ba262d3647d4"}
Oct 30 19:49:21.396529 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:21.395929    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx" event={"ID":"a70f60f8-d58b-4db3-a200-4958351a5e47","Type":"ContainerStarted","Data":"8befd2a16a12e7d868ab9d93b08878d914553b0ce363c3c0888203ba9c6c20da"}
Oct 30 19:49:21.396529 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:21.396318    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"
Oct 30 19:49:21.418222 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:21.418174    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx" podStartSLOduration=2.418158017 podStartE2EDuration="2.418158017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:21.415110508 +0000 UTC m=+1040.330717365" watchObservedRunningTime="2026-10-30 19:49:21.418158017 +0000 UTC m=+1040.333764991"
Oct 30 19:49:23.135617 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.135581    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"]
Oct 30 19:49:23.139575 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.139556    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"
Oct 30 19:49:23.145535 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.145497    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-59e21a\""
Oct 30 19:49:23.145662 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.145644    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-59e21a-dockercfg-mqwlq\""
Oct 30 19:49:23.148199 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.148156    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"]
Oct 30 19:49:23.193563 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.193531    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"]
Oct 30 19:49:23.198980 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.198404    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"
Oct 30 19:49:23.225584 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.225551    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"]
Oct 30 19:49:23.243792 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.243756    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96057e18-15ef-4b05-b46d-6899653ae6ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf\" (UID: \"96057e18-15ef-4b05-b46d-6899653ae6ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"
Oct 30 19:49:23.243957 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.243901    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j9hj\" (UniqueName: \"kubernetes.io/projected/2b0530a0-02ec-4fa1-a987-403de3ae09db-kube-api-access-6j9hj\") pod \"payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t\" (UID: \"2b0530a0-02ec-4fa1-a987-403de3ae09db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"
Oct 30 19:49:23.244004 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.243973    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x89t8\" (UniqueName: \"kubernetes.io/projected/96057e18-15ef-4b05-b46d-6899653ae6ba-kube-api-access-x89t8\") pod \"payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf\" (UID: \"96057e18-15ef-4b05-b46d-6899653ae6ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"
Oct 30 19:49:23.244039 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.244032    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0530a0-02ec-4fa1-a987-403de3ae09db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t\" (UID: \"2b0530a0-02ec-4fa1-a987-403de3ae09db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"
Oct 30 19:49:23.348066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.347999    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j9hj\" (UniqueName: \"kubernetes.io/projected/2b0530a0-02ec-4fa1-a987-403de3ae09db-kube-api-access-6j9hj\") pod \"payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t\" (UID: \"2b0530a0-02ec-4fa1-a987-403de3ae09db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"
Oct 30 19:49:23.348259 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.348145    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x89t8\" (UniqueName: \"kubernetes.io/projected/96057e18-15ef-4b05-b46d-6899653ae6ba-kube-api-access-x89t8\") pod \"payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf\" (UID: \"96057e18-15ef-4b05-b46d-6899653ae6ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"
Oct 30 19:49:23.348259 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.348226    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0530a0-02ec-4fa1-a987-403de3ae09db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t\" (UID: \"2b0530a0-02ec-4fa1-a987-403de3ae09db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"
Oct 30 19:49:23.348365 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.348347    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96057e18-15ef-4b05-b46d-6899653ae6ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf\" (UID: \"96057e18-15ef-4b05-b46d-6899653ae6ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"
Oct 30 19:49:23.378070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.372387    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96057e18-15ef-4b05-b46d-6899653ae6ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf\" (UID: \"96057e18-15ef-4b05-b46d-6899653ae6ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"
Oct 30 19:49:23.419123 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.389772    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0530a0-02ec-4fa1-a987-403de3ae09db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t\" (UID: \"2b0530a0-02ec-4fa1-a987-403de3ae09db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"
Oct 30 19:49:23.424774 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.421746    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j9hj\" (UniqueName: \"kubernetes.io/projected/2b0530a0-02ec-4fa1-a987-403de3ae09db-kube-api-access-6j9hj\") pod \"payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t\" (UID: \"2b0530a0-02ec-4fa1-a987-403de3ae09db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"
Oct 30 19:49:23.424774 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.422350    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x89t8\" (UniqueName: \"kubernetes.io/projected/96057e18-15ef-4b05-b46d-6899653ae6ba-kube-api-access-x89t8\") pod \"payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf\" (UID: \"96057e18-15ef-4b05-b46d-6899653ae6ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"
Oct 30 19:49:23.469309 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.469277    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"
Oct 30 19:49:23.532091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.525343    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"
Oct 30 19:49:23.890966 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:49:23.889428    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96057e18_15ef_4b05_b46d_6899653ae6ba.slice/crio-5cfa5ad6900fd667705d48af12dce9da8d8783fb69a4c38954f586f8d517930b WatchSource:0}: Error finding container 5cfa5ad6900fd667705d48af12dce9da8d8783fb69a4c38954f586f8d517930b: Status 404 returned error can't find the container with id 5cfa5ad6900fd667705d48af12dce9da8d8783fb69a4c38954f586f8d517930b
Oct 30 19:49:23.925327 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.925248    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"]
Oct 30 19:49:23.946696 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:23.946655    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"]
Oct 30 19:49:24.465308 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:24.465257    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t" event={"ID":"2b0530a0-02ec-4fa1-a987-403de3ae09db","Type":"ContainerStarted","Data":"6702387d11989989a7c0462f163778cdda722f218c173724c11d0ceadbe58f71"}
Oct 30 19:49:24.465791 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:24.465337    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t" event={"ID":"2b0530a0-02ec-4fa1-a987-403de3ae09db","Type":"ContainerStarted","Data":"f258afaeef507f452e58c6a24c752dc13cff4926b0642236c55c46d69a42b1ab"}
Oct 30 19:49:24.465791 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:24.465367    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"
Oct 30 19:49:24.467477 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:24.467432    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf" event={"ID":"96057e18-15ef-4b05-b46d-6899653ae6ba","Type":"ContainerStarted","Data":"55d894dc07562dff053cc944e63d39a47b17eb89794d11588d60658b6b8f5673"}
Oct 30 19:49:24.467685 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:24.467522    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf" event={"ID":"96057e18-15ef-4b05-b46d-6899653ae6ba","Type":"ContainerStarted","Data":"5cfa5ad6900fd667705d48af12dce9da8d8783fb69a4c38954f586f8d517930b"}
Oct 30 19:49:24.467756 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:24.467735    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"
Oct 30 19:49:24.487912 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:24.487869    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t" podStartSLOduration=1.487854958 podStartE2EDuration="1.487854958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:24.486166533 +0000 UTC m=+1043.401773389" watchObservedRunningTime="2026-10-30 19:49:24.487854958 +0000 UTC m=+1043.403461814"
Oct 30 19:49:24.507939 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:24.507893    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf" podStartSLOduration=1.5078798660000001 podStartE2EDuration="1.507879866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:24.506454823 +0000 UTC m=+1043.422061679" watchObservedRunningTime="2026-10-30 19:49:24.507879866 +0000 UTC m=+1043.423486721"
Oct 30 19:49:26.285340 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:26.285308    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"]
Oct 30 19:49:26.286119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:26.286083    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25" podUID="7d7a5e10-e02b-482b-81ac-21be2160fad9" containerName="maas-api" containerID="cri-o://d8b84bb19900d620bac3cb9faf0562c9b8f56960227e4acbd227cd2f70f99a8d" gracePeriod=30
Oct 30 19:49:26.290583 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:26.290465    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25" podUID="7d7a5e10-e02b-482b-81ac-21be2160fad9" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.86:8443/health\": dial tcp 10.133.0.86:8443: connect: connection refused"
Oct 30 19:49:26.895117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:26.889121    6030 generic.go:356] "Generic (PLEG): container finished" podID="7d7a5e10-e02b-482b-81ac-21be2160fad9" containerID="d8b84bb19900d620bac3cb9faf0562c9b8f56960227e4acbd227cd2f70f99a8d" exitCode=0
Oct 30 19:49:26.895117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:26.889204    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25" event={"ID":"7d7a5e10-e02b-482b-81ac-21be2160fad9","Type":"ContainerDied","Data":"d8b84bb19900d620bac3cb9faf0562c9b8f56960227e4acbd227cd2f70f99a8d"}
Oct 30 19:49:26.951127 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:26.951088    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:49:27.037074 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:27.037007    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7a5e10-e02b-482b-81ac-21be2160fad9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7a5e10-e02b-482b-81ac-21be2160fad9-openshift-service-ca\") pod \"7d7a5e10-e02b-482b-81ac-21be2160fad9\" (UID: \"7d7a5e10-e02b-482b-81ac-21be2160fad9\") "
Oct 30 19:49:27.037263 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:27.037151    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d7a5e10-e02b-482b-81ac-21be2160fad9-kube-api-access-6g9vx\" (UniqueName: \"kubernetes.io/projected/7d7a5e10-e02b-482b-81ac-21be2160fad9-kube-api-access-6g9vx\") pod \"7d7a5e10-e02b-482b-81ac-21be2160fad9\" (UID: \"7d7a5e10-e02b-482b-81ac-21be2160fad9\") "
Oct 30 19:49:27.037263 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:27.037192    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7a5e10-e02b-482b-81ac-21be2160fad9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7a5e10-e02b-482b-81ac-21be2160fad9-maas-api-tls\") pod \"7d7a5e10-e02b-482b-81ac-21be2160fad9\" (UID: \"7d7a5e10-e02b-482b-81ac-21be2160fad9\") "
Oct 30 19:49:27.037263 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:27.037230    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7a5e10-e02b-482b-81ac-21be2160fad9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7a5e10-e02b-482b-81ac-21be2160fad9-metrics-certs\") pod \"7d7a5e10-e02b-482b-81ac-21be2160fad9\" (UID: \"7d7a5e10-e02b-482b-81ac-21be2160fad9\") "
Oct 30 19:49:27.037454 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:27.037427    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7a5e10-e02b-482b-81ac-21be2160fad9-openshift-service-ca" pod "7d7a5e10-e02b-482b-81ac-21be2160fad9" (UID: "7d7a5e10-e02b-482b-81ac-21be2160fad9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:27.037852 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:27.037805    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7a5e10-e02b-482b-81ac-21be2160fad9-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.039738 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:27.039705    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7a5e10-e02b-482b-81ac-21be2160fad9-kube-api-access-6g9vx" pod "7d7a5e10-e02b-482b-81ac-21be2160fad9" (UID: "7d7a5e10-e02b-482b-81ac-21be2160fad9"). InnerVolumeSpecName "kube-api-access-6g9vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:27.040343 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:27.040318    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7a5e10-e02b-482b-81ac-21be2160fad9-maas-api-tls" pod "7d7a5e10-e02b-482b-81ac-21be2160fad9" (UID: "7d7a5e10-e02b-482b-81ac-21be2160fad9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:27.040434 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:27.040385    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7a5e10-e02b-482b-81ac-21be2160fad9-metrics-certs" pod "7d7a5e10-e02b-482b-81ac-21be2160fad9" (UID: "7d7a5e10-e02b-482b-81ac-21be2160fad9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:27.159119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:27.145953    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g9vx\" (UniqueName: \"kubernetes.io/projected/7d7a5e10-e02b-482b-81ac-21be2160fad9-kube-api-access-6g9vx\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.159119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:27.145995    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7a5e10-e02b-482b-81ac-21be2160fad9-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.159119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:27.146010    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7a5e10-e02b-482b-81ac-21be2160fad9-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.902427 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:27.902400    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"
Oct 30 19:49:27.902915 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:27.902405    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25" event={"ID":"7d7a5e10-e02b-482b-81ac-21be2160fad9","Type":"ContainerDied","Data":"ed970821000d50e8634abc9f39c01e1ecc348ef7a9575c3ac73e40eecdd0712c"}
Oct 30 19:49:27.902915 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:27.902541    6030 scope.go:122] "RemoveContainer" containerID="d8b84bb19900d620bac3cb9faf0562c9b8f56960227e4acbd227cd2f70f99a8d"
Oct 30 19:49:27.924291 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:27.924263    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"]
Oct 30 19:49:27.930926 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:27.930899    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2761c7-7999cf6469-psn25"]
Oct 30 19:49:28.303742 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.303651    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"]
Oct 30 19:49:28.307598 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.304517    6030 cpu_manager.go:414] "containerMap: removing container" podUID="7d7a5e10-e02b-482b-81ac-21be2160fad9" containerName="maas-api"
Oct 30 19:49:28.307598 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.304539    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7a5e10-e02b-482b-81ac-21be2160fad9" containerName="maas-api"
Oct 30 19:49:28.307598 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.304743    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7a5e10-e02b-482b-81ac-21be2160fad9" containerName="maas-api"
Oct 30 19:49:28.313545 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.313517    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"]
Oct 30 19:49:28.313682 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.313665    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:49:28.319744 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.319723    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:49:28.319871 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.319777    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1e3723-serving-cert\""
Oct 30 19:49:28.320222 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.320207    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d892r\""
Oct 30 19:49:28.320300 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.320273    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:49:28.320793 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.320776    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:49:28.362223 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.362194    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-maas-api-tls\") pod \"maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m\" (UID: \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:49:28.362374 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.362262    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-metrics-certs\") pod \"maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m\" (UID: \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:49:28.362429 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.362406    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-openshift-service-ca\") pod \"maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m\" (UID: \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:49:28.362466 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.362450    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vlk2\" (UniqueName: \"kubernetes.io/projected/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-kube-api-access-9vlk2\") pod \"maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m\" (UID: \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:49:28.469272 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.469230    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-openshift-service-ca\") pod \"maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m\" (UID: \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:49:28.485070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.469533    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vlk2\" (UniqueName: \"kubernetes.io/projected/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-kube-api-access-9vlk2\") pod \"maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m\" (UID: \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:49:28.485070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.469675    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-maas-api-tls\") pod \"maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m\" (UID: \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:49:28.485070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.469707    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-metrics-certs\") pod \"maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m\" (UID: \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:49:28.487330 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.487297    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-openshift-service-ca\") pod \"maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m\" (UID: \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:49:28.533963 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.533930    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-metrics-certs\") pod \"maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m\" (UID: \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:49:28.534906 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.534878    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-maas-api-tls\") pod \"maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m\" (UID: \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:49:28.535889 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.535839    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vlk2\" (UniqueName: \"kubernetes.io/projected/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-kube-api-access-9vlk2\") pod \"maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m\" (UID: \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:49:28.629073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.629025    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:49:28.941142 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:28.941065    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"]
Oct 30 19:49:28.945596 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:49:28.945567    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e3cf00_165f_4cd0_b98e_4713a8ffbda1.slice/crio-47a7509335fea8732c10fac9dfa4fb4568893fa758f558ab628a02b9c3910f35 WatchSource:0}: Error finding container 47a7509335fea8732c10fac9dfa4fb4568893fa758f558ab628a02b9c3910f35: Status 404 returned error can't find the container with id 47a7509335fea8732c10fac9dfa4fb4568893fa758f558ab628a02b9c3910f35
Oct 30 19:49:29.219115 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:29.215761    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d7a5e10-e02b-482b-81ac-21be2160fad9" path="/var/lib/kubelet/pods/7d7a5e10-e02b-482b-81ac-21be2160fad9/volumes"
Oct 30 19:49:29.919507 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:29.919475    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m" event={"ID":"39e3cf00-165f-4cd0-b98e-4713a8ffbda1","Type":"ContainerStarted","Data":"14a03f89046bdc855709d18502a6dd610297771136a5dac82c3bc5e3adc3619d"}
Oct 30 19:49:29.919685 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:29.919529    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m" event={"ID":"39e3cf00-165f-4cd0-b98e-4713a8ffbda1","Type":"ContainerStarted","Data":"47a7509335fea8732c10fac9dfa4fb4568893fa758f558ab628a02b9c3910f35"}
Oct 30 19:49:29.919685 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:29.919550    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:49:29.942379 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:29.942324    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m" podStartSLOduration=1.448598947 podStartE2EDuration="1.942306306s" totalImagesPullingTime="493.707359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:29.941488383 +0000 UTC m=+1048.857095240" watchObservedRunningTime="2026-10-30 19:49:29.942306306 +0000 UTC m=+1048.857913166"
Oct 30 19:49:31.024998 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:31.024962    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"]
Oct 30 19:49:31.028072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:31.028033    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590ea3fb-tlczp"]
Oct 30 19:49:31.206687 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:31.206657    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a18142c9-b841-4aca-9480-ad43ab4a0625" path="/var/lib/kubelet/pods/a18142c9-b841-4aca-9480-ad43ab4a0625/volumes"
Oct 30 19:49:31.399702 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:31.399674    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"
Oct 30 19:49:32.026242 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.026202    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"]
Oct 30 19:49:32.030982 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.030949    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:49:32.037861 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.037830    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-59e21a-serving-cert\""
Oct 30 19:49:32.056837 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.056805    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"]
Oct 30 19:49:32.110757 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.110711    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n7xn\" (UniqueName: \"kubernetes.io/projected/cac65540-89ac-4591-b288-14f8ac45be99-kube-api-access-5n7xn\") pod \"maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz\" (UID: \"cac65540-89ac-4591-b288-14f8ac45be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:49:32.110935 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.110891    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac65540-89ac-4591-b288-14f8ac45be99-maas-api-tls\") pod \"maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz\" (UID: \"cac65540-89ac-4591-b288-14f8ac45be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:49:32.111124 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.111104    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac65540-89ac-4591-b288-14f8ac45be99-metrics-certs\") pod \"maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz\" (UID: \"cac65540-89ac-4591-b288-14f8ac45be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:49:32.111187 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.111175    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac65540-89ac-4591-b288-14f8ac45be99-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz\" (UID: \"cac65540-89ac-4591-b288-14f8ac45be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:49:32.213082 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.212945    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac65540-89ac-4591-b288-14f8ac45be99-metrics-certs\") pod \"maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz\" (UID: \"cac65540-89ac-4591-b288-14f8ac45be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:49:32.213082 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.213063    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac65540-89ac-4591-b288-14f8ac45be99-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz\" (UID: \"cac65540-89ac-4591-b288-14f8ac45be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:49:32.213379 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.213176    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n7xn\" (UniqueName: \"kubernetes.io/projected/cac65540-89ac-4591-b288-14f8ac45be99-kube-api-access-5n7xn\") pod \"maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz\" (UID: \"cac65540-89ac-4591-b288-14f8ac45be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:49:32.213379 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.213267    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac65540-89ac-4591-b288-14f8ac45be99-maas-api-tls\") pod \"maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz\" (UID: \"cac65540-89ac-4591-b288-14f8ac45be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:49:32.214142 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.214109    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac65540-89ac-4591-b288-14f8ac45be99-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz\" (UID: \"cac65540-89ac-4591-b288-14f8ac45be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:49:32.217558 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.217527    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac65540-89ac-4591-b288-14f8ac45be99-maas-api-tls\") pod \"maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz\" (UID: \"cac65540-89ac-4591-b288-14f8ac45be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:49:32.217700 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.217583    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac65540-89ac-4591-b288-14f8ac45be99-metrics-certs\") pod \"maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz\" (UID: \"cac65540-89ac-4591-b288-14f8ac45be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:49:32.231896 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.231862    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n7xn\" (UniqueName: \"kubernetes.io/projected/cac65540-89ac-4591-b288-14f8ac45be99-kube-api-access-5n7xn\") pod \"maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz\" (UID: \"cac65540-89ac-4591-b288-14f8ac45be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:49:32.351120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.349150    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:49:32.427319 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.427283    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"
Oct 30 19:49:32.677069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.673447    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"]
Oct 30 19:49:32.677069 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:49:32.674459    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac65540_89ac_4591_b288_14f8ac45be99.slice/crio-4790d64c396216c61ecafc384675ab34048b8658d4522b774e214ba8e5fe7a32 WatchSource:0}: Error finding container 4790d64c396216c61ecafc384675ab34048b8658d4522b774e214ba8e5fe7a32: Status 404 returned error can't find the container with id 4790d64c396216c61ecafc384675ab34048b8658d4522b774e214ba8e5fe7a32
Oct 30 19:49:32.966767 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:32.966655    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz" event={"ID":"cac65540-89ac-4591-b288-14f8ac45be99","Type":"ContainerStarted","Data":"4790d64c396216c61ecafc384675ab34048b8658d4522b774e214ba8e5fe7a32"}
Oct 30 19:49:34.008755 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:34.008710    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz" event={"ID":"cac65540-89ac-4591-b288-14f8ac45be99","Type":"ContainerStarted","Data":"b49b23d80fcff213d355968360d120a841a5569d899a72198026ae1c9c79a114"}
Oct 30 19:49:34.009807 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:34.009783    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:49:34.080539 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:34.080475    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz" podStartSLOduration=2.621399298 podStartE2EDuration="3.080455178s" totalImagesPullingTime="459.05588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:34.064535072 +0000 UTC m=+1052.980141930" watchObservedRunningTime="2026-10-30 19:49:34.080455178 +0000 UTC m=+1052.996062035"
Oct 30 19:49:34.375536 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:49:34.375500    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:49:35.551069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:35.550285    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"
Oct 30 19:49:35.555066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:35.551782    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"
Oct 30 19:49:35.917896 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:35.917538    6030 patch_prober.go:29] interesting pod/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:49:35.917896 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:35.917599    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx" podUID="45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:35.935339 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:35.934241    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:49:38.079451 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:38.079339    6030 generic.go:356] "Generic (PLEG): container finished" podID="45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" containerID="2d85ef91684daaf90ad19f582411a2b74d4ff3f815567cabc3f0a3d0877d5a5b" exitCode=0
Oct 30 19:49:38.080174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:38.080016    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx" event={"ID":"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8","Type":"ContainerDied","Data":"2d85ef91684daaf90ad19f582411a2b74d4ff3f815567cabc3f0a3d0877d5a5b"}
Oct 30 19:49:38.081341 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:38.081321    6030 scope.go:122] "RemoveContainer" containerID="2d85ef91684daaf90ad19f582411a2b74d4ff3f815567cabc3f0a3d0877d5a5b"
Oct 30 19:49:38.085172 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:38.084068    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:49:38.085172 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:38.084164    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:49:38.085172 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:38.084214    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:49:39.119900 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:39.119858    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx" event={"ID":"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8","Type":"ContainerStarted","Data":"36555afb27906c5993066b65d501ccab0914430b7c230476c4a49e15c1c851d9"}
Oct 30 19:49:39.232285 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:39.232257    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"]
Oct 30 19:49:39.235157 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:39.235137    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"
Oct 30 19:49:39.244393 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:39.243037    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-040623-dockercfg-t626x\""
Oct 30 19:49:39.244887 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:39.244861    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-040623\""
Oct 30 19:49:39.248389 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:39.248369    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"]
Oct 30 19:49:39.348073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:39.330127    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c35f7e3-cd45-4bea-bf41-ab50c131dffc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g\" (UID: \"0c35f7e3-cd45-4bea-bf41-ab50c131dffc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"
Oct 30 19:49:39.348073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:39.330249    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phxmp\" (UniqueName: \"kubernetes.io/projected/0c35f7e3-cd45-4bea-bf41-ab50c131dffc-kube-api-access-phxmp\") pod \"payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g\" (UID: \"0c35f7e3-cd45-4bea-bf41-ab50c131dffc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"
Oct 30 19:49:39.477069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:39.476650    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phxmp\" (UniqueName: \"kubernetes.io/projected/0c35f7e3-cd45-4bea-bf41-ab50c131dffc-kube-api-access-phxmp\") pod \"payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g\" (UID: \"0c35f7e3-cd45-4bea-bf41-ab50c131dffc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"
Oct 30 19:49:39.477069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:39.476934    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c35f7e3-cd45-4bea-bf41-ab50c131dffc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g\" (UID: \"0c35f7e3-cd45-4bea-bf41-ab50c131dffc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"
Oct 30 19:49:39.499070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:39.477834    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c35f7e3-cd45-4bea-bf41-ab50c131dffc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g\" (UID: \"0c35f7e3-cd45-4bea-bf41-ab50c131dffc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"
Oct 30 19:49:39.512072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:39.506638    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:49:39.512072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:39.506793    6030 patch_prober.go:29] interesting pod/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:49:39.512072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:39.506837    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx" podUID="45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:49:39.539988 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:39.539957    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phxmp\" (UniqueName: \"kubernetes.io/projected/0c35f7e3-cd45-4bea-bf41-ab50c131dffc-kube-api-access-phxmp\") pod \"payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g\" (UID: \"0c35f7e3-cd45-4bea-bf41-ab50c131dffc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"
Oct 30 19:49:39.556068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:39.551174    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"
Oct 30 19:49:39.936312 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:39.935667    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"]
Oct 30 19:49:39.937863 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:49:39.937836    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c35f7e3_cd45_4bea_bf41_ab50c131dffc.slice/crio-0b5947b270f80611d761f26d86679e7d331ab158f4752d7a1e042d332d8e9cc9 WatchSource:0}: Error finding container 0b5947b270f80611d761f26d86679e7d331ab158f4752d7a1e042d332d8e9cc9: Status 404 returned error can't find the container with id 0b5947b270f80611d761f26d86679e7d331ab158f4752d7a1e042d332d8e9cc9
Oct 30 19:49:40.128242 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:40.128208    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g" event={"ID":"0c35f7e3-cd45-4bea-bf41-ab50c131dffc","Type":"ContainerStarted","Data":"8648215d457e6eda0cb30cb27df030ffdbf44bd3460580e4e1cecd581d560aaf"}
Oct 30 19:49:40.128688 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:40.128257    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g" event={"ID":"0c35f7e3-cd45-4bea-bf41-ab50c131dffc","Type":"ContainerStarted","Data":"0b5947b270f80611d761f26d86679e7d331ab158f4752d7a1e042d332d8e9cc9"}
Oct 30 19:49:40.128688 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:40.128548    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"
Oct 30 19:49:40.150604 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:40.150563    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g" podStartSLOduration=1.150549394 podStartE2EDuration="1.150549394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:40.149876629 +0000 UTC m=+1059.065483489" watchObservedRunningTime="2026-10-30 19:49:40.150549394 +0000 UTC m=+1059.066156250"
Oct 30 19:49:40.455785 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:40.455750    6030 patch_prober.go:29] interesting pod/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:49:40.455969 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:40.455809    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx" podUID="45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:49:41.064139 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:41.064106    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:49:41.455002 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:41.454965    6030 patch_prober.go:29] interesting pod/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:49:41.455462 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:41.455034    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx" podUID="45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:49:42.455314 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:42.455281    6030 patch_prober.go:29] interesting pod/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:49:42.455681 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:42.455336    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx" podUID="45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:49:43.455743 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:43.455709    6030 patch_prober.go:29] interesting pod/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:49:43.456166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:43.455759    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx" podUID="45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:49:44.455898 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:44.455861    6030 patch_prober.go:29] interesting pod/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:49:44.456303 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:44.455931    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx" podUID="45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:49:45.459235 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:45.459203    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:49:45.459687 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:45.459539    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:49:45.460421 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:45.460399    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:49:48.927107 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:48.927060    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"]
Oct 30 19:49:48.931180 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:48.931160    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"
Oct 30 19:49:48.940033 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:48.940007    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4f2325\""
Oct 30 19:49:48.941272 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:48.941249    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4f2325-dockercfg-wcmbp\""
Oct 30 19:49:48.943232 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:48.943208    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"]
Oct 30 19:49:48.947241 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:48.947222    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"
Oct 30 19:49:48.967109 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:48.967080    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a2d13b0-c920-4e1d-8a36-950f4532c235-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph\" (UID: \"1a2d13b0-c920-4e1d-8a36-950f4532c235\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"
Oct 30 19:49:48.967275 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:48.967246    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbftj\" (UniqueName: \"kubernetes.io/projected/74ede63b-4f53-4e61-ab60-4b79ccb3101d-kube-api-access-tbftj\") pod \"payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf\" (UID: \"74ede63b-4f53-4e61-ab60-4b79ccb3101d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"
Oct 30 19:49:48.967401 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:48.967385    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ede63b-4f53-4e61-ab60-4b79ccb3101d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf\" (UID: \"74ede63b-4f53-4e61-ab60-4b79ccb3101d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"
Oct 30 19:49:48.967511 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:48.967496    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdkpk\" (UniqueName: \"kubernetes.io/projected/1a2d13b0-c920-4e1d-8a36-950f4532c235-kube-api-access-cdkpk\") pod \"payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph\" (UID: \"1a2d13b0-c920-4e1d-8a36-950f4532c235\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"
Oct 30 19:49:48.968136 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:48.968116    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"]
Oct 30 19:49:48.971481 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:48.971457    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"]
Oct 30 19:49:49.081071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:49.076553    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbftj\" (UniqueName: \"kubernetes.io/projected/74ede63b-4f53-4e61-ab60-4b79ccb3101d-kube-api-access-tbftj\") pod \"payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf\" (UID: \"74ede63b-4f53-4e61-ab60-4b79ccb3101d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"
Oct 30 19:49:49.081071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:49.076703    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ede63b-4f53-4e61-ab60-4b79ccb3101d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf\" (UID: \"74ede63b-4f53-4e61-ab60-4b79ccb3101d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"
Oct 30 19:49:49.081071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:49.076809    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdkpk\" (UniqueName: \"kubernetes.io/projected/1a2d13b0-c920-4e1d-8a36-950f4532c235-kube-api-access-cdkpk\") pod \"payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph\" (UID: \"1a2d13b0-c920-4e1d-8a36-950f4532c235\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"
Oct 30 19:49:49.081071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:49.076889    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a2d13b0-c920-4e1d-8a36-950f4532c235-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph\" (UID: \"1a2d13b0-c920-4e1d-8a36-950f4532c235\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"
Oct 30 19:49:49.081071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:49.077898    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a2d13b0-c920-4e1d-8a36-950f4532c235-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph\" (UID: \"1a2d13b0-c920-4e1d-8a36-950f4532c235\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"
Oct 30 19:49:49.130086 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:49.119005    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ede63b-4f53-4e61-ab60-4b79ccb3101d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf\" (UID: \"74ede63b-4f53-4e61-ab60-4b79ccb3101d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"
Oct 30 19:49:49.156108 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:49.154721    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbftj\" (UniqueName: \"kubernetes.io/projected/74ede63b-4f53-4e61-ab60-4b79ccb3101d-kube-api-access-tbftj\") pod \"payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf\" (UID: \"74ede63b-4f53-4e61-ab60-4b79ccb3101d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"
Oct 30 19:49:49.165069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:49.156667    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdkpk\" (UniqueName: \"kubernetes.io/projected/1a2d13b0-c920-4e1d-8a36-950f4532c235-kube-api-access-cdkpk\") pod \"payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph\" (UID: \"1a2d13b0-c920-4e1d-8a36-950f4532c235\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"
Oct 30 19:49:49.248355 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:49.246461    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"
Oct 30 19:49:49.261305 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:49.261268    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"
Oct 30 19:49:49.678815 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:49.678422    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"]
Oct 30 19:49:49.728102 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:49.713436    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"]
Oct 30 19:49:50.330167 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:50.330060    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph" event={"ID":"1a2d13b0-c920-4e1d-8a36-950f4532c235","Type":"ContainerStarted","Data":"0ce3f341a4f37975066408580f75e7b88b8392c74421c14a6d49838c42612260"}
Oct 30 19:49:50.330167 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:50.330125    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph" event={"ID":"1a2d13b0-c920-4e1d-8a36-950f4532c235","Type":"ContainerStarted","Data":"2a6695547cd8fb8655e22ecc4e0bd5805ae43435b8bf2e0db22838e7034dfc54"}
Oct 30 19:49:50.330167 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:50.330154    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"
Oct 30 19:49:50.333113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:50.332578    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf" event={"ID":"74ede63b-4f53-4e61-ab60-4b79ccb3101d","Type":"ContainerStarted","Data":"a4bb907def79b77fd836d4ff86c2b650a37366a7957b9db18db2634f48af2c14"}
Oct 30 19:49:50.333113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:50.332630    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf" event={"ID":"74ede63b-4f53-4e61-ab60-4b79ccb3101d","Type":"ContainerStarted","Data":"bb5e5a644fe9dcd0efe1342e5a484a49cf1b99565a062817883e7d113052d9dc"}
Oct 30 19:49:50.333113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:50.332660    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"
Oct 30 19:49:50.353327 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:50.353269    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph" podStartSLOduration=2.353256315 podStartE2EDuration="2.353256315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:50.350215228 +0000 UTC m=+1069.265822084" watchObservedRunningTime="2026-10-30 19:49:50.353256315 +0000 UTC m=+1069.268863170"
Oct 30 19:49:50.379991 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:50.379943    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf" podStartSLOduration=2.379929192 podStartE2EDuration="2.379929192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:50.377733364 +0000 UTC m=+1069.293340220" watchObservedRunningTime="2026-10-30 19:49:50.379929192 +0000 UTC m=+1069.295536113"
Oct 30 19:49:51.141077 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.141005    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"
Oct 30 19:49:51.622123 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.622018    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"]
Oct 30 19:49:51.626675 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.626651    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:49:51.644185 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.641834    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-040623-serving-cert\""
Oct 30 19:49:51.648864 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.648838    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"]
Oct 30 19:49:51.712424 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.712386    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1feae8e-f123-45b9-a52b-86457fabe1ec-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-040623-d5446d769-57kwk\" (UID: \"c1feae8e-f123-45b9-a52b-86457fabe1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:49:51.712581 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.712437    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7wmp\" (UniqueName: \"kubernetes.io/projected/c1feae8e-f123-45b9-a52b-86457fabe1ec-kube-api-access-g7wmp\") pod \"maas-api-e2e-ipp-b-040623-d5446d769-57kwk\" (UID: \"c1feae8e-f123-45b9-a52b-86457fabe1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:49:51.712581 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.712549    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1feae8e-f123-45b9-a52b-86457fabe1ec-metrics-certs\") pod \"maas-api-e2e-ipp-b-040623-d5446d769-57kwk\" (UID: \"c1feae8e-f123-45b9-a52b-86457fabe1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:49:51.712581 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.712577    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1feae8e-f123-45b9-a52b-86457fabe1ec-maas-api-tls\") pod \"maas-api-e2e-ipp-b-040623-d5446d769-57kwk\" (UID: \"c1feae8e-f123-45b9-a52b-86457fabe1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:49:51.814469 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.814430    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1feae8e-f123-45b9-a52b-86457fabe1ec-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-040623-d5446d769-57kwk\" (UID: \"c1feae8e-f123-45b9-a52b-86457fabe1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:49:51.814469 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.814475    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7wmp\" (UniqueName: \"kubernetes.io/projected/c1feae8e-f123-45b9-a52b-86457fabe1ec-kube-api-access-g7wmp\") pod \"maas-api-e2e-ipp-b-040623-d5446d769-57kwk\" (UID: \"c1feae8e-f123-45b9-a52b-86457fabe1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:49:51.814706 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.814525    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1feae8e-f123-45b9-a52b-86457fabe1ec-metrics-certs\") pod \"maas-api-e2e-ipp-b-040623-d5446d769-57kwk\" (UID: \"c1feae8e-f123-45b9-a52b-86457fabe1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:49:51.814706 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.814544    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1feae8e-f123-45b9-a52b-86457fabe1ec-maas-api-tls\") pod \"maas-api-e2e-ipp-b-040623-d5446d769-57kwk\" (UID: \"c1feae8e-f123-45b9-a52b-86457fabe1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:49:51.815292 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.815268    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1feae8e-f123-45b9-a52b-86457fabe1ec-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-040623-d5446d769-57kwk\" (UID: \"c1feae8e-f123-45b9-a52b-86457fabe1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:49:51.817324 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.817295    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1feae8e-f123-45b9-a52b-86457fabe1ec-metrics-certs\") pod \"maas-api-e2e-ipp-b-040623-d5446d769-57kwk\" (UID: \"c1feae8e-f123-45b9-a52b-86457fabe1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:49:51.817472 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.817330    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1feae8e-f123-45b9-a52b-86457fabe1ec-maas-api-tls\") pod \"maas-api-e2e-ipp-b-040623-d5446d769-57kwk\" (UID: \"c1feae8e-f123-45b9-a52b-86457fabe1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:49:51.875260 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.875021    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7wmp\" (UniqueName: \"kubernetes.io/projected/c1feae8e-f123-45b9-a52b-86457fabe1ec-kube-api-access-g7wmp\") pod \"maas-api-e2e-ipp-b-040623-d5446d769-57kwk\" (UID: \"c1feae8e-f123-45b9-a52b-86457fabe1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:49:51.951068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:51.949276    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:49:52.199158 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:52.199075    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"]
Oct 30 19:49:52.221586 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:52.219975    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"]
Oct 30 19:49:52.283704 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:52.283663    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"]
Oct 30 19:49:52.307067 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:49:52.305358    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1feae8e_f123_45b9_a52b_86457fabe1ec.slice/crio-dd434ced30b06f8b64fdb9e649e1d277a6ede480410674c0fa74cf015da37572 WatchSource:0}: Error finding container dd434ced30b06f8b64fdb9e649e1d277a6ede480410674c0fa74cf015da37572: Status 404 returned error can't find the container with id dd434ced30b06f8b64fdb9e649e1d277a6ede480410674c0fa74cf015da37572
Oct 30 19:49:52.375912 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:52.375700    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk" event={"ID":"c1feae8e-f123-45b9-a52b-86457fabe1ec","Type":"ContainerStarted","Data":"dd434ced30b06f8b64fdb9e649e1d277a6ede480410674c0fa74cf015da37572"}
Oct 30 19:49:53.045168 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:53.045128    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"]
Oct 30 19:49:53.052025 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:53.052001    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eae23df-jbqgc"]
Oct 30 19:49:53.218070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:53.209681    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fbd5646-a264-465a-8159-2e01c0526294" path="/var/lib/kubelet/pods/1fbd5646-a264-465a-8159-2e01c0526294/volumes"
Oct 30 19:49:53.412110 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:53.405099    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk" event={"ID":"c1feae8e-f123-45b9-a52b-86457fabe1ec","Type":"ContainerStarted","Data":"2c42a7215e63d71eadf7a04766a545c544f8607fcd87010e25d60da155ee9442"}
Oct 30 19:49:53.412110 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:53.406514    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:49:53.436224 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:53.436165    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk" podStartSLOduration=1.918866896 podStartE2EDuration="2.436150114s" totalImagesPullingTime="517.283218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:53.432470381 +0000 UTC m=+1072.348077250" watchObservedRunningTime="2026-10-30 19:49:53.436150114 +0000 UTC m=+1072.351756971"
Oct 30 19:49:57.200178 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.199583    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t" podUID="2b0530a0-02ec-4fa1-a987-403de3ae09db" containerName="payload-processing" containerID="cri-o://6702387d11989989a7c0462f163778cdda722f218c173724c11d0ceadbe58f71" gracePeriod=25
Oct 30 19:49:57.221323 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.221294    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf" podUID="96057e18-15ef-4b05-b46d-6899653ae6ba" containerName="payload-pre-processing" containerID="cri-o://55d894dc07562dff053cc944e63d39a47b17eb89794d11588d60658b6b8f5673" gracePeriod=25
Oct 30 19:49:57.454161 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.453919    6030 generic.go:356] "Generic (PLEG): container finished" podID="2b0530a0-02ec-4fa1-a987-403de3ae09db" containerID="6702387d11989989a7c0462f163778cdda722f218c173724c11d0ceadbe58f71" exitCode=0
Oct 30 19:49:57.454161 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.454007    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t" event={"ID":"2b0530a0-02ec-4fa1-a987-403de3ae09db","Type":"ContainerDied","Data":"6702387d11989989a7c0462f163778cdda722f218c173724c11d0ceadbe58f71"}
Oct 30 19:49:57.455643 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.455612    6030 generic.go:356] "Generic (PLEG): container finished" podID="96057e18-15ef-4b05-b46d-6899653ae6ba" containerID="55d894dc07562dff053cc944e63d39a47b17eb89794d11588d60658b6b8f5673" exitCode=0
Oct 30 19:49:57.455763 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.455729    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf" event={"ID":"96057e18-15ef-4b05-b46d-6899653ae6ba","Type":"ContainerDied","Data":"55d894dc07562dff053cc944e63d39a47b17eb89794d11588d60658b6b8f5673"}
Oct 30 19:49:57.619369 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.619342    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"
Oct 30 19:49:57.659303 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.659278    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"
Oct 30 19:49:57.662725 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.662701    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96057e18-15ef-4b05-b46d-6899653ae6ba-kube-api-access-x89t8\" (UniqueName: \"kubernetes.io/projected/96057e18-15ef-4b05-b46d-6899653ae6ba-kube-api-access-x89t8\") pod \"96057e18-15ef-4b05-b46d-6899653ae6ba\" (UID: \"96057e18-15ef-4b05-b46d-6899653ae6ba\") "
Oct 30 19:49:57.662817 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.662761    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96057e18-15ef-4b05-b46d-6899653ae6ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96057e18-15ef-4b05-b46d-6899653ae6ba-plugins-config-volume\") pod \"96057e18-15ef-4b05-b46d-6899653ae6ba\" (UID: \"96057e18-15ef-4b05-b46d-6899653ae6ba\") "
Oct 30 19:49:57.663400 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.663376    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96057e18-15ef-4b05-b46d-6899653ae6ba-plugins-config-volume" pod "96057e18-15ef-4b05-b46d-6899653ae6ba" (UID: "96057e18-15ef-4b05-b46d-6899653ae6ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:57.665444 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.665420    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96057e18-15ef-4b05-b46d-6899653ae6ba-kube-api-access-x89t8" pod "96057e18-15ef-4b05-b46d-6899653ae6ba" (UID: "96057e18-15ef-4b05-b46d-6899653ae6ba"). InnerVolumeSpecName "kube-api-access-x89t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:57.764240 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.764153    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b0530a0-02ec-4fa1-a987-403de3ae09db-kube-api-access-6j9hj\" (UniqueName: \"kubernetes.io/projected/2b0530a0-02ec-4fa1-a987-403de3ae09db-kube-api-access-6j9hj\") pod \"2b0530a0-02ec-4fa1-a987-403de3ae09db\" (UID: \"2b0530a0-02ec-4fa1-a987-403de3ae09db\") "
Oct 30 19:49:57.764390 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.764286    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b0530a0-02ec-4fa1-a987-403de3ae09db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0530a0-02ec-4fa1-a987-403de3ae09db-plugins-config-volume\") pod \"2b0530a0-02ec-4fa1-a987-403de3ae09db\" (UID: \"2b0530a0-02ec-4fa1-a987-403de3ae09db\") "
Oct 30 19:49:57.764642 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.764629    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x89t8\" (UniqueName: \"kubernetes.io/projected/96057e18-15ef-4b05-b46d-6899653ae6ba-kube-api-access-x89t8\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:57.764692 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.764645    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96057e18-15ef-4b05-b46d-6899653ae6ba-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:57.764692 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.764643    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0530a0-02ec-4fa1-a987-403de3ae09db-plugins-config-volume" pod "2b0530a0-02ec-4fa1-a987-403de3ae09db" (UID: "2b0530a0-02ec-4fa1-a987-403de3ae09db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:57.766588 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.766564    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0530a0-02ec-4fa1-a987-403de3ae09db-kube-api-access-6j9hj" pod "2b0530a0-02ec-4fa1-a987-403de3ae09db" (UID: "2b0530a0-02ec-4fa1-a987-403de3ae09db"). InnerVolumeSpecName "kube-api-access-6j9hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:57.865727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.865692    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j9hj\" (UniqueName: \"kubernetes.io/projected/2b0530a0-02ec-4fa1-a987-403de3ae09db-kube-api-access-6j9hj\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:57.865727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:57.865722    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0530a0-02ec-4fa1-a987-403de3ae09db-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:58.463418 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:58.463390    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"
Oct 30 19:49:58.463418 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:58.463407    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t" event={"ID":"2b0530a0-02ec-4fa1-a987-403de3ae09db","Type":"ContainerDied","Data":"f258afaeef507f452e58c6a24c752dc13cff4926b0642236c55c46d69a42b1ab"}
Oct 30 19:49:58.463902 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:58.463469    6030 scope.go:122] "RemoveContainer" containerID="6702387d11989989a7c0462f163778cdda722f218c173724c11d0ceadbe58f71"
Oct 30 19:49:58.465018 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:58.464994    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf" event={"ID":"96057e18-15ef-4b05-b46d-6899653ae6ba","Type":"ContainerDied","Data":"5cfa5ad6900fd667705d48af12dce9da8d8783fb69a4c38954f586f8d517930b"}
Oct 30 19:49:58.465134 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:58.465020    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"
Oct 30 19:49:58.549763 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:58.539103    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"]
Oct 30 19:49:58.550174 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:58.550149    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-849b8bbd95-82slf"]
Oct 30 19:49:58.550371 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:58.540469    6030 scope.go:122] "RemoveContainer" containerID="55d894dc07562dff053cc944e63d39a47b17eb89794d11588d60658b6b8f5673"
Oct 30 19:49:58.564342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:58.563319    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"]
Oct 30 19:49:58.597072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:58.575101    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59e21a-75b5d75c8d-rbx6t"]
Oct 30 19:49:59.202400 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:59.202373    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b0530a0-02ec-4fa1-a987-403de3ae09db" path="/var/lib/kubelet/pods/2b0530a0-02ec-4fa1-a987-403de3ae09db/volumes"
Oct 30 19:49:59.202783 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:49:59.202770    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96057e18-15ef-4b05-b46d-6899653ae6ba" path="/var/lib/kubelet/pods/96057e18-15ef-4b05-b46d-6899653ae6ba/volumes"
Oct 30 19:50:00.440751 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:00.440713    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:50:01.345901 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.345872    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"
Oct 30 19:50:01.346086 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.345948    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"
Oct 30 19:50:01.725553 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.725511    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"]
Oct 30 19:50:01.726405 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.726379    6030 cpu_manager.go:414] "containerMap: removing container" podUID="2b0530a0-02ec-4fa1-a987-403de3ae09db" containerName="payload-processing"
Oct 30 19:50:01.726405 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.726401    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b0530a0-02ec-4fa1-a987-403de3ae09db" containerName="payload-processing"
Oct 30 19:50:01.726716 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.726444    6030 cpu_manager.go:414] "containerMap: removing container" podUID="96057e18-15ef-4b05-b46d-6899653ae6ba" containerName="payload-pre-processing"
Oct 30 19:50:01.726716 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.726452    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96057e18-15ef-4b05-b46d-6899653ae6ba" containerName="payload-pre-processing"
Oct 30 19:50:01.727183 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.727003    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b0530a0-02ec-4fa1-a987-403de3ae09db" containerName="payload-processing"
Oct 30 19:50:01.727183 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.727060    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="96057e18-15ef-4b05-b46d-6899653ae6ba" containerName="payload-pre-processing"
Oct 30 19:50:01.737091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.737034    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:50:01.745548 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.745514    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"]
Oct 30 19:50:01.748712 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.748683    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4f2325-serving-cert\""
Oct 30 19:50:01.800417 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.800387    6030 scope.go:122] "RemoveContainer" containerID="bfecb12069982eaa713050e9fb52e5c3d093e764a373e3cf110df4efcd201eee"
Oct 30 19:50:01.852276 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.852252    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzfnc\" (UniqueName: \"kubernetes.io/projected/46f60b19-53a4-4b56-8a23-20501f978935-kube-api-access-pzfnc\") pod \"maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt\" (UID: \"46f60b19-53a4-4b56-8a23-20501f978935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:50:01.852384 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.852297    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f60b19-53a4-4b56-8a23-20501f978935-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt\" (UID: \"46f60b19-53a4-4b56-8a23-20501f978935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:50:01.852572 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.852548    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46f60b19-53a4-4b56-8a23-20501f978935-metrics-certs\") pod \"maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt\" (UID: \"46f60b19-53a4-4b56-8a23-20501f978935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:50:01.852678 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.852660    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f60b19-53a4-4b56-8a23-20501f978935-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt\" (UID: \"46f60b19-53a4-4b56-8a23-20501f978935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:50:01.958794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.954353    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46f60b19-53a4-4b56-8a23-20501f978935-metrics-certs\") pod \"maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt\" (UID: \"46f60b19-53a4-4b56-8a23-20501f978935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:50:01.958794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.954450    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f60b19-53a4-4b56-8a23-20501f978935-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt\" (UID: \"46f60b19-53a4-4b56-8a23-20501f978935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:50:01.958794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.954589    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzfnc\" (UniqueName: \"kubernetes.io/projected/46f60b19-53a4-4b56-8a23-20501f978935-kube-api-access-pzfnc\") pod \"maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt\" (UID: \"46f60b19-53a4-4b56-8a23-20501f978935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:50:01.958794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.954628    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f60b19-53a4-4b56-8a23-20501f978935-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt\" (UID: \"46f60b19-53a4-4b56-8a23-20501f978935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:50:01.973215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.973134    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f60b19-53a4-4b56-8a23-20501f978935-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt\" (UID: \"46f60b19-53a4-4b56-8a23-20501f978935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:50:01.995667 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.995579    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f60b19-53a4-4b56-8a23-20501f978935-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt\" (UID: \"46f60b19-53a4-4b56-8a23-20501f978935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:50:01.996511 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:01.996472    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzfnc\" (UniqueName: \"kubernetes.io/projected/46f60b19-53a4-4b56-8a23-20501f978935-kube-api-access-pzfnc\") pod \"maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt\" (UID: \"46f60b19-53a4-4b56-8a23-20501f978935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:50:02.019656 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:02.019625    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46f60b19-53a4-4b56-8a23-20501f978935-metrics-certs\") pod \"maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt\" (UID: \"46f60b19-53a4-4b56-8a23-20501f978935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:50:02.061028 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:02.060985    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:50:02.434714 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:02.434557    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"]
Oct 30 19:50:02.592691 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:02.592653    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt" event={"ID":"46f60b19-53a4-4b56-8a23-20501f978935","Type":"ContainerStarted","Data":"b0c8d77a24c05b24090949717050081b0f0757617c9e7949fff5a16190774612"}
Oct 30 19:50:03.601214 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:03.601182    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt" event={"ID":"46f60b19-53a4-4b56-8a23-20501f978935","Type":"ContainerStarted","Data":"60e7dac58ba33e547f055a804101a162b010c65e55e349804c5a1404900e3665"}
Oct 30 19:50:03.601673 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:03.601258    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:50:03.602823 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:03.602800    6030 generic.go:356] "Generic (PLEG): container finished" podID="8de060ca-04d8-4c9e-aff5-78d3dbbc0199" containerID="c580b2b203e5f0e9ac68df6aade37d1facda218fd802604e2ea2593f2732c2bc" exitCode=0
Oct 30 19:50:03.602945 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:03.602865    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj" event={"ID":"8de060ca-04d8-4c9e-aff5-78d3dbbc0199","Type":"ContainerDied","Data":"c580b2b203e5f0e9ac68df6aade37d1facda218fd802604e2ea2593f2732c2bc"}
Oct 30 19:50:03.603238 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:03.603224    6030 scope.go:122] "RemoveContainer" containerID="c580b2b203e5f0e9ac68df6aade37d1facda218fd802604e2ea2593f2732c2bc"
Oct 30 19:50:03.604643 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:03.604612    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:50:03.612029 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:03.605320    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:50:03.612029 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:03.605476    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:50:03.663376 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:03.663324    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt" podStartSLOduration=2.08265572 podStartE2EDuration="2.663309665s" totalImagesPullingTime="580.653945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:03.660902461 +0000 UTC m=+1082.576509319" watchObservedRunningTime="2026-10-30 19:50:03.663309665 +0000 UTC m=+1082.578916527"
Oct 30 19:50:04.008806 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.008774    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"]
Oct 30 19:50:04.013224 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.013201    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"
Oct 30 19:50:04.025733 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.025701    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-59e21a\""
Oct 30 19:50:04.026256 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.026223    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-59e21a-dockercfg-rjr44\""
Oct 30 19:50:04.030105 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.030085    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"]
Oct 30 19:50:04.146070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.130494    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv42h\" (UniqueName: \"kubernetes.io/projected/14294872-1aaf-4b7d-b2ab-f80fbf7db165-kube-api-access-vv42h\") pod \"payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8\" (UID: \"14294872-1aaf-4b7d-b2ab-f80fbf7db165\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"
Oct 30 19:50:04.146070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.130646    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14294872-1aaf-4b7d-b2ab-f80fbf7db165-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8\" (UID: \"14294872-1aaf-4b7d-b2ab-f80fbf7db165\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"
Oct 30 19:50:04.146070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.134580    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"]
Oct 30 19:50:04.146070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.141288    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"
Oct 30 19:50:04.156136 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.156105    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"]
Oct 30 19:50:04.268132 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.263638    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv42h\" (UniqueName: \"kubernetes.io/projected/14294872-1aaf-4b7d-b2ab-f80fbf7db165-kube-api-access-vv42h\") pod \"payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8\" (UID: \"14294872-1aaf-4b7d-b2ab-f80fbf7db165\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"
Oct 30 19:50:04.333770 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.333729    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14294872-1aaf-4b7d-b2ab-f80fbf7db165-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8\" (UID: \"14294872-1aaf-4b7d-b2ab-f80fbf7db165\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"
Oct 30 19:50:04.334128 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.334011    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkrjz\" (UniqueName: \"kubernetes.io/projected/8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5-kube-api-access-rkrjz\") pod \"payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc\" (UID: \"8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"
Oct 30 19:50:04.355089 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.339078    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14294872-1aaf-4b7d-b2ab-f80fbf7db165-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8\" (UID: \"14294872-1aaf-4b7d-b2ab-f80fbf7db165\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"
Oct 30 19:50:04.355089 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.351218    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc\" (UID: \"8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"
Oct 30 19:50:04.401321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.379331    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv42h\" (UniqueName: \"kubernetes.io/projected/14294872-1aaf-4b7d-b2ab-f80fbf7db165-kube-api-access-vv42h\") pod \"payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8\" (UID: \"14294872-1aaf-4b7d-b2ab-f80fbf7db165\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"
Oct 30 19:50:04.463078 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.462476    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkrjz\" (UniqueName: \"kubernetes.io/projected/8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5-kube-api-access-rkrjz\") pod \"payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc\" (UID: \"8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"
Oct 30 19:50:04.463078 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.462740    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc\" (UID: \"8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"
Oct 30 19:50:04.464356 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.464218    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc\" (UID: \"8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"
Oct 30 19:50:04.521137 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:50:04.519779    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:50:04.522696 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.522634    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkrjz\" (UniqueName: \"kubernetes.io/projected/8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5-kube-api-access-rkrjz\") pod \"payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc\" (UID: \"8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"
Oct 30 19:50:04.611507 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.611470    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj" event={"ID":"8de060ca-04d8-4c9e-aff5-78d3dbbc0199","Type":"ContainerStarted","Data":"3f940f3d46e522c8876bedb87919cefab88c9e573f3af29248eeb2ebbdbd595c"}
Oct 30 19:50:04.628815 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.628785    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"
Oct 30 19:50:04.761739 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.761707    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"
Oct 30 19:50:04.961671 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:04.961637    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"]
Oct 30 19:50:05.019760 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:05.019733    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:50:05.020651 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:05.020626    6030 patch_prober.go:29] interesting pod/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:50:05.020703 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:05.020675    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj" podUID="8de060ca-04d8-4c9e-aff5-78d3dbbc0199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:50:05.053249 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:05.053224    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"]
Oct 30 19:50:05.058698 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:05.058602    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f67ec5-rc9z6"]
Oct 30 19:50:05.234893 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:05.234863    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9a625e6-791a-4492-9cb5-621e5c52f41a" path="/var/lib/kubelet/pods/a9a625e6-791a-4492-9cb5-621e5c52f41a/volumes"
Oct 30 19:50:05.235258 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:05.235238    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"]
Oct 30 19:50:05.389192 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:05.389151    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:50:05.620032 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:05.619938    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc" event={"ID":"8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5","Type":"ContainerStarted","Data":"32e5a408bfc272d8ffdaa9d479e6ba2a96f3bc17539c2b5c9455829cdeeef408"}
Oct 30 19:50:05.620534 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:05.620507    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc" event={"ID":"8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5","Type":"ContainerStarted","Data":"b29714bc2182ba8eeb8094f9e9d2049ef47e576f7ded61bffc74f0a8576df3e5"}
Oct 30 19:50:05.620667 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:05.620656    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"
Oct 30 19:50:05.621832 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:05.621810    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8" event={"ID":"14294872-1aaf-4b7d-b2ab-f80fbf7db165","Type":"ContainerStarted","Data":"66a141525b994f847211674c755ebcec6b1ee15cbff7c2df48833479eeb753ee"}
Oct 30 19:50:05.621926 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:05.621847    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8" event={"ID":"14294872-1aaf-4b7d-b2ab-f80fbf7db165","Type":"ContainerStarted","Data":"6c2973231590bf988a8aa50084a0305c5f0a7d6f3e05503df926272958a6a5b1"}
Oct 30 19:50:05.642030 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:05.641983    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc" podStartSLOduration=1.6419693130000002 podStartE2EDuration="1.641969313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:05.640489928 +0000 UTC m=+1084.556096786" watchObservedRunningTime="2026-10-30 19:50:05.641969313 +0000 UTC m=+1084.557576170"
Oct 30 19:50:05.668081 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:05.668000    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8" podStartSLOduration=2.667983037 podStartE2EDuration="2.667983037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:05.662580478 +0000 UTC m=+1084.578187334" watchObservedRunningTime="2026-10-30 19:50:05.667983037 +0000 UTC m=+1084.583589894"
Oct 30 19:50:06.017571 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:06.017527    6030 patch_prober.go:29] interesting pod/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:50:06.017788 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:06.017594    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj" podUID="8de060ca-04d8-4c9e-aff5-78d3dbbc0199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:50:06.629176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:06.629142    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"
Oct 30 19:50:07.017881 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:07.017801    6030 patch_prober.go:29] interesting pod/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:50:07.017881 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:07.017855    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj" podUID="8de060ca-04d8-4c9e-aff5-78d3dbbc0199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:50:08.017772 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:08.017736    6030 patch_prober.go:29] interesting pod/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:50:08.018239 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:08.017798    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj" podUID="8de060ca-04d8-4c9e-aff5-78d3dbbc0199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:50:09.030025 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:09.024491    6030 patch_prober.go:29] interesting pod/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:50:09.030025 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:09.024558    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj" podUID="8de060ca-04d8-4c9e-aff5-78d3dbbc0199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:50:09.617909 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:09.617582    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:50:10.405741 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:10.405679    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:50:10.407282 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:10.407258    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:50:13.401260 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:13.401217    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"]
Oct 30 19:50:16.631238 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:16.631194    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"
Oct 30 19:50:17.638808 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:17.638773    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"
Oct 30 19:50:18.402483 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:18.402444    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g" podUID="0c35f7e3-cd45-4bea-bf41-ab50c131dffc" containerName="payload-pre-processing" containerID="cri-o://8648215d457e6eda0cb30cb27df030ffdbf44bd3460580e4e1cecd581d560aaf" gracePeriod=25
Oct 30 19:50:18.718779 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:18.718456    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"
Oct 30 19:50:18.767329 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:18.767295    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c35f7e3-cd45-4bea-bf41-ab50c131dffc-kube-api-access-phxmp\" (UniqueName: \"kubernetes.io/projected/0c35f7e3-cd45-4bea-bf41-ab50c131dffc-kube-api-access-phxmp\") pod \"0c35f7e3-cd45-4bea-bf41-ab50c131dffc\" (UID: \"0c35f7e3-cd45-4bea-bf41-ab50c131dffc\") "
Oct 30 19:50:18.767561 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:18.767545    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c35f7e3-cd45-4bea-bf41-ab50c131dffc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c35f7e3-cd45-4bea-bf41-ab50c131dffc-plugins-config-volume\") pod \"0c35f7e3-cd45-4bea-bf41-ab50c131dffc\" (UID: \"0c35f7e3-cd45-4bea-bf41-ab50c131dffc\") "
Oct 30 19:50:18.768132 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:18.768099    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c35f7e3-cd45-4bea-bf41-ab50c131dffc-plugins-config-volume" pod "0c35f7e3-cd45-4bea-bf41-ab50c131dffc" (UID: "0c35f7e3-cd45-4bea-bf41-ab50c131dffc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:50:18.769712 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:18.769683    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c35f7e3-cd45-4bea-bf41-ab50c131dffc-kube-api-access-phxmp" pod "0c35f7e3-cd45-4bea-bf41-ab50c131dffc" (UID: "0c35f7e3-cd45-4bea-bf41-ab50c131dffc"). InnerVolumeSpecName "kube-api-access-phxmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:18.772966 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:18.772934    6030 generic.go:356] "Generic (PLEG): container finished" podID="0c35f7e3-cd45-4bea-bf41-ab50c131dffc" containerID="8648215d457e6eda0cb30cb27df030ffdbf44bd3460580e4e1cecd581d560aaf" exitCode=0
Oct 30 19:50:18.773111 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:18.772989    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"
Oct 30 19:50:18.773111 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:18.773014    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g" event={"ID":"0c35f7e3-cd45-4bea-bf41-ab50c131dffc","Type":"ContainerDied","Data":"8648215d457e6eda0cb30cb27df030ffdbf44bd3460580e4e1cecd581d560aaf"}
Oct 30 19:50:18.773226 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:18.773135    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g" event={"ID":"0c35f7e3-cd45-4bea-bf41-ab50c131dffc","Type":"ContainerDied","Data":"0b5947b270f80611d761f26d86679e7d331ab158f4752d7a1e042d332d8e9cc9"}
Oct 30 19:50:18.773226 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:18.773168    6030 scope.go:122] "RemoveContainer" containerID="8648215d457e6eda0cb30cb27df030ffdbf44bd3460580e4e1cecd581d560aaf"
Oct 30 19:50:18.796300 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:18.796277    6030 scope.go:122] "RemoveContainer" containerID="8648215d457e6eda0cb30cb27df030ffdbf44bd3460580e4e1cecd581d560aaf"
Oct 30 19:50:18.796696 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:50:18.796663    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8648215d457e6eda0cb30cb27df030ffdbf44bd3460580e4e1cecd581d560aaf\": container with ID starting with 8648215d457e6eda0cb30cb27df030ffdbf44bd3460580e4e1cecd581d560aaf not found: ID does not exist" containerID="8648215d457e6eda0cb30cb27df030ffdbf44bd3460580e4e1cecd581d560aaf"
Oct 30 19:50:18.796808 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:18.796707    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8648215d457e6eda0cb30cb27df030ffdbf44bd3460580e4e1cecd581d560aaf"} err="failed to get container status \"8648215d457e6eda0cb30cb27df030ffdbf44bd3460580e4e1cecd581d560aaf\": rpc error: code = NotFound desc = could not find container \"8648215d457e6eda0cb30cb27df030ffdbf44bd3460580e4e1cecd581d560aaf\": container with ID starting with 8648215d457e6eda0cb30cb27df030ffdbf44bd3460580e4e1cecd581d560aaf not found: ID does not exist"
Oct 30 19:50:18.831122 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:18.831088    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"]
Oct 30 19:50:18.832314 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:50:18.832279    6030 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c35f7e3_cd45_4bea_bf41_ab50c131dffc.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:50:18.960152 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:18.902184    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c35f7e3-cd45-4bea-bf41-ab50c131dffc-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:18.960152 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:18.902217    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phxmp\" (UniqueName: \"kubernetes.io/projected/0c35f7e3-cd45-4bea-bf41-ab50c131dffc-kube-api-access-phxmp\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:18.995070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:18.986672    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-5f54648f55-hx52g"]
Oct 30 19:50:19.201747 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:19.201716    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c35f7e3-cd45-4bea-bf41-ab50c131dffc" path="/var/lib/kubelet/pods/0c35f7e3-cd45-4bea-bf41-ab50c131dffc/volumes"
Oct 30 19:50:21.340894 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.340860    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"]
Oct 30 19:50:21.341783 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.341764    6030 cpu_manager.go:414] "containerMap: removing container" podUID="0c35f7e3-cd45-4bea-bf41-ab50c131dffc" containerName="payload-pre-processing"
Oct 30 19:50:21.341845 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.341786    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c35f7e3-cd45-4bea-bf41-ab50c131dffc" containerName="payload-pre-processing"
Oct 30 19:50:21.341955 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.341943    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c35f7e3-cd45-4bea-bf41-ab50c131dffc" containerName="payload-pre-processing"
Oct 30 19:50:21.364226 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.364194    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.366280 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.366252    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"]
Oct 30 19:50:21.387670 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.387573    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e907e4-openshift-default-dockercfg-lkqh5\""
Oct 30 19:50:21.447902 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.447870    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-podinfo\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.447902 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.447910    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-workload-socket\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.448136 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.447937    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-credential-socket\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.448180 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.448156    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-token\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.448245 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.448227    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwks6\" (UniqueName: \"kubernetes.io/projected/e4869c19-e834-4312-a859-ca5af2c9cf63-kube-api-access-hwks6\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.448298 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.448282    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-workload-certs\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.448365 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.448340    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4869c19-e834-4312-a859-ca5af2c9cf63-istiod-ca-cert\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.448407 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.448393    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-envoy\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.448510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.448498    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-data\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.550817 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.550546    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-podinfo\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.550817 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.550610    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-workload-socket\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.550817 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.550658    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-credential-socket\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.550817 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.550777    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-token\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.551221 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.550861    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwks6\" (UniqueName: \"kubernetes.io/projected/e4869c19-e834-4312-a859-ca5af2c9cf63-kube-api-access-hwks6\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.551221 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.550942    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-workload-certs\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.551221 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.550983    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4869c19-e834-4312-a859-ca5af2c9cf63-istiod-ca-cert\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.551221 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.551033    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-envoy\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.551221 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.551137    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-data\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.555082 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.551700    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-data\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.555082 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.552382    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-workload-certs\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.562073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.559484    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-workload-socket\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.565456 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.565421    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-credential-socket\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.588088 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.568237    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4869c19-e834-4312-a859-ca5af2c9cf63-istiod-ca-cert\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.588088 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.576754    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-podinfo\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.588088 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.579324    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-envoy\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.598123 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.593954    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwks6\" (UniqueName: \"kubernetes.io/projected/e4869c19-e834-4312-a859-ca5af2c9cf63-kube-api-access-hwks6\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.598123 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.595263    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-token\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-799mq\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:21.694060 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:21.694017    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:22.055996 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:22.055967    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"]
Oct 30 19:50:22.069862 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:22.069832    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"]
Oct 30 19:50:22.070798 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:50:22.070774    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4869c19_e834_4312_a859_ca5af2c9cf63.slice/crio-07895e1344529043a3e47276ee04d9a5977bd7293011c6260867dd6955737969 WatchSource:0}: Error finding container 07895e1344529043a3e47276ee04d9a5977bd7293011c6260867dd6955737969: Status 404 returned error can't find the container with id 07895e1344529043a3e47276ee04d9a5977bd7293011c6260867dd6955737969
Oct 30 19:50:22.081384 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:22.081343    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:50:22.081499 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:22.081428    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:50:22.081499 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:22.081468    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:50:22.083018 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:22.082998    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c7c9036-jjwzm"]
Oct 30 19:50:22.862088 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:22.859254    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq" event={"ID":"e4869c19-e834-4312-a859-ca5af2c9cf63","Type":"ContainerStarted","Data":"c24a253ab8a3d3d8b95d310a6b64dd46c4bc88777c01fef9479254f82804da51"}
Oct 30 19:50:22.862088 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:22.859360    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq" event={"ID":"e4869c19-e834-4312-a859-ca5af2c9cf63","Type":"ContainerStarted","Data":"07895e1344529043a3e47276ee04d9a5977bd7293011c6260867dd6955737969"}
Oct 30 19:50:22.929085 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:22.917013    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq" podStartSLOduration=1.916990162 podStartE2EDuration="1.916990162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:22.909522456 +0000 UTC m=+1101.825129317" watchObservedRunningTime="2026-10-30 19:50:22.916990162 +0000 UTC m=+1101.832597019"
Oct 30 19:50:23.046091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:23.039700    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"]
Oct 30 19:50:23.133095 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:23.130773    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"]
Oct 30 19:50:23.219986 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:23.219950    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7501f0e7-9aea-46de-8455-ab837f1cc08f" path="/var/lib/kubelet/pods/7501f0e7-9aea-46de-8455-ab837f1cc08f/volumes"
Oct 30 19:50:23.698079 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:23.695974    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:23.698079 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:23.696164    6030 patch_prober.go:29] interesting pod/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:50:23.698079 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:23.696228    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq" podUID="e4869c19-e834-4312-a859-ca5af2c9cf63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:50:24.696072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:24.695510    6030 patch_prober.go:29] interesting pod/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:50:24.696072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:24.695569    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq" podUID="e4869c19-e834-4312-a859-ca5af2c9cf63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:50:25.694533 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:25.694492    6030 patch_prober.go:29] interesting pod/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:50:25.694757 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:25.694578    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq" podUID="e4869c19-e834-4312-a859-ca5af2c9cf63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:50:26.694498 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:26.694454    6030 patch_prober.go:29] interesting pod/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:50:26.695081 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:26.694537    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq" podUID="e4869c19-e834-4312-a859-ca5af2c9cf63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:50:27.694846 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:27.694447    6030 patch_prober.go:29] interesting pod/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:50:27.694846 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:27.694510    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq" podUID="e4869c19-e834-4312-a859-ca5af2c9cf63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:50:28.044786 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.040449    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph" podUID="1a2d13b0-c920-4e1d-8a36-950f4532c235" containerName="payload-processing" containerID="cri-o://0ce3f341a4f37975066408580f75e7b88b8392c74421c14a6d49838c42612260" gracePeriod=25
Oct 30 19:50:28.132287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.132232    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf" podUID="74ede63b-4f53-4e61-ab60-4b79ccb3101d" containerName="payload-pre-processing" containerID="cri-o://a4bb907def79b77fd836d4ff86c2b650a37366a7957b9db18db2634f48af2c14" gracePeriod=25
Oct 30 19:50:28.492729 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.492692    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"]
Oct 30 19:50:28.504573 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.504542    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"
Oct 30 19:50:28.505959 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.505168    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"]
Oct 30 19:50:28.513992 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.513962    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-040623\""
Oct 30 19:50:28.514213 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.514194    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-040623-dockercfg-rsp5n\""
Oct 30 19:50:28.515102 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.515078    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"
Oct 30 19:50:28.542237 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.542205    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"
Oct 30 19:50:28.577035 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.576993    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ede63b-4f53-4e61-ab60-4b79ccb3101d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ede63b-4f53-4e61-ab60-4b79ccb3101d-plugins-config-volume\") pod \"74ede63b-4f53-4e61-ab60-4b79ccb3101d\" (UID: \"74ede63b-4f53-4e61-ab60-4b79ccb3101d\") "
Oct 30 19:50:28.577275 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.577095    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a2d13b0-c920-4e1d-8a36-950f4532c235-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a2d13b0-c920-4e1d-8a36-950f4532c235-plugins-config-volume\") pod \"1a2d13b0-c920-4e1d-8a36-950f4532c235\" (UID: \"1a2d13b0-c920-4e1d-8a36-950f4532c235\") "
Oct 30 19:50:28.577275 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.577145    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a2d13b0-c920-4e1d-8a36-950f4532c235-kube-api-access-cdkpk\" (UniqueName: \"kubernetes.io/projected/1a2d13b0-c920-4e1d-8a36-950f4532c235-kube-api-access-cdkpk\") pod \"1a2d13b0-c920-4e1d-8a36-950f4532c235\" (UID: \"1a2d13b0-c920-4e1d-8a36-950f4532c235\") "
Oct 30 19:50:28.577400 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.577331    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ede63b-4f53-4e61-ab60-4b79ccb3101d-kube-api-access-tbftj\" (UniqueName: \"kubernetes.io/projected/74ede63b-4f53-4e61-ab60-4b79ccb3101d-kube-api-access-tbftj\") pod \"74ede63b-4f53-4e61-ab60-4b79ccb3101d\" (UID: \"74ede63b-4f53-4e61-ab60-4b79ccb3101d\") "
Oct 30 19:50:28.577998 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.577955    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr65n\" (UniqueName: \"kubernetes.io/projected/54a2ed94-fdfe-41ac-aced-bc1f2c326d06-kube-api-access-nr65n\") pod \"payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf\" (UID: \"54a2ed94-fdfe-41ac-aced-bc1f2c326d06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"
Oct 30 19:50:28.578146 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.578029    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a2ed94-fdfe-41ac-aced-bc1f2c326d06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf\" (UID: \"54a2ed94-fdfe-41ac-aced-bc1f2c326d06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"
Oct 30 19:50:28.578861 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.578819    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ede63b-4f53-4e61-ab60-4b79ccb3101d-plugins-config-volume" pod "74ede63b-4f53-4e61-ab60-4b79ccb3101d" (UID: "74ede63b-4f53-4e61-ab60-4b79ccb3101d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:50:28.579385 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.579359    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2d13b0-c920-4e1d-8a36-950f4532c235-plugins-config-volume" pod "1a2d13b0-c920-4e1d-8a36-950f4532c235" (UID: "1a2d13b0-c920-4e1d-8a36-950f4532c235"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:50:28.580127 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.580099    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"]
Oct 30 19:50:28.580839 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.580820    6030 cpu_manager.go:414] "containerMap: removing container" podUID="74ede63b-4f53-4e61-ab60-4b79ccb3101d" containerName="payload-pre-processing"
Oct 30 19:50:28.580936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.580841    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ede63b-4f53-4e61-ab60-4b79ccb3101d" containerName="payload-pre-processing"
Oct 30 19:50:28.580936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.580918    6030 cpu_manager.go:414] "containerMap: removing container" podUID="1a2d13b0-c920-4e1d-8a36-950f4532c235" containerName="payload-processing"
Oct 30 19:50:28.580936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.580927    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2d13b0-c920-4e1d-8a36-950f4532c235" containerName="payload-processing"
Oct 30 19:50:28.581117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.581106    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2d13b0-c920-4e1d-8a36-950f4532c235" containerName="payload-processing"
Oct 30 19:50:28.581168 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.581124    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ede63b-4f53-4e61-ab60-4b79ccb3101d" containerName="payload-pre-processing"
Oct 30 19:50:28.608695 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.583528    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2d13b0-c920-4e1d-8a36-950f4532c235-kube-api-access-cdkpk" pod "1a2d13b0-c920-4e1d-8a36-950f4532c235" (UID: "1a2d13b0-c920-4e1d-8a36-950f4532c235"). InnerVolumeSpecName "kube-api-access-cdkpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:28.608695 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.583588    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ede63b-4f53-4e61-ab60-4b79ccb3101d-kube-api-access-tbftj" pod "74ede63b-4f53-4e61-ab60-4b79ccb3101d" (UID: "74ede63b-4f53-4e61-ab60-4b79ccb3101d"). InnerVolumeSpecName "kube-api-access-tbftj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:28.608695 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.588852    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"
Oct 30 19:50:28.683084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.680966    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l7cv\" (UniqueName: \"kubernetes.io/projected/3919c09d-9f26-4233-a9b9-6f38d6e39970-kube-api-access-2l7cv\") pod \"payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q\" (UID: \"3919c09d-9f26-4233-a9b9-6f38d6e39970\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"
Oct 30 19:50:28.683084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.681071    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr65n\" (UniqueName: \"kubernetes.io/projected/54a2ed94-fdfe-41ac-aced-bc1f2c326d06-kube-api-access-nr65n\") pod \"payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf\" (UID: \"54a2ed94-fdfe-41ac-aced-bc1f2c326d06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"
Oct 30 19:50:28.683084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.681125    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a2ed94-fdfe-41ac-aced-bc1f2c326d06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf\" (UID: \"54a2ed94-fdfe-41ac-aced-bc1f2c326d06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"
Oct 30 19:50:28.683084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.681228    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3919c09d-9f26-4233-a9b9-6f38d6e39970-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q\" (UID: \"3919c09d-9f26-4233-a9b9-6f38d6e39970\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"
Oct 30 19:50:28.683084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.681332    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbftj\" (UniqueName: \"kubernetes.io/projected/74ede63b-4f53-4e61-ab60-4b79ccb3101d-kube-api-access-tbftj\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:28.683084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.681351    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ede63b-4f53-4e61-ab60-4b79ccb3101d-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:28.683084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.681365    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a2d13b0-c920-4e1d-8a36-950f4532c235-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:28.683084 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.681386    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdkpk\" (UniqueName: \"kubernetes.io/projected/1a2d13b0-c920-4e1d-8a36-950f4532c235-kube-api-access-cdkpk\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:28.706726 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.706689    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a2ed94-fdfe-41ac-aced-bc1f2c326d06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf\" (UID: \"54a2ed94-fdfe-41ac-aced-bc1f2c326d06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"
Oct 30 19:50:28.734398 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.734145    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"]
Oct 30 19:50:28.757365 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.757281    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr65n\" (UniqueName: \"kubernetes.io/projected/54a2ed94-fdfe-41ac-aced-bc1f2c326d06-kube-api-access-nr65n\") pod \"payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf\" (UID: \"54a2ed94-fdfe-41ac-aced-bc1f2c326d06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"
Oct 30 19:50:28.785238 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.783160    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l7cv\" (UniqueName: \"kubernetes.io/projected/3919c09d-9f26-4233-a9b9-6f38d6e39970-kube-api-access-2l7cv\") pod \"payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q\" (UID: \"3919c09d-9f26-4233-a9b9-6f38d6e39970\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"
Oct 30 19:50:28.785238 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.783304    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3919c09d-9f26-4233-a9b9-6f38d6e39970-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q\" (UID: \"3919c09d-9f26-4233-a9b9-6f38d6e39970\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"
Oct 30 19:50:28.832903 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.830112    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l7cv\" (UniqueName: \"kubernetes.io/projected/3919c09d-9f26-4233-a9b9-6f38d6e39970-kube-api-access-2l7cv\") pod \"payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q\" (UID: \"3919c09d-9f26-4233-a9b9-6f38d6e39970\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"
Oct 30 19:50:28.837146 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.696169    6030 patch_prober.go:29] interesting pod/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:50:28.848071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.837374    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"
Oct 30 19:50:28.863707 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.837437    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq" podUID="e4869c19-e834-4312-a859-ca5af2c9cf63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:50:28.864512 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.864478    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3919c09d-9f26-4233-a9b9-6f38d6e39970-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q\" (UID: \"3919c09d-9f26-4233-a9b9-6f38d6e39970\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"
Oct 30 19:50:28.924294 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.924157    6030 generic.go:356] "Generic (PLEG): container finished" podID="1a2d13b0-c920-4e1d-8a36-950f4532c235" containerID="0ce3f341a4f37975066408580f75e7b88b8392c74421c14a6d49838c42612260" exitCode=0
Oct 30 19:50:28.924294 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.924237    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"
Oct 30 19:50:28.924294 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.924240    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph" event={"ID":"1a2d13b0-c920-4e1d-8a36-950f4532c235","Type":"ContainerDied","Data":"0ce3f341a4f37975066408580f75e7b88b8392c74421c14a6d49838c42612260"}
Oct 30 19:50:28.924765 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.924310    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph" event={"ID":"1a2d13b0-c920-4e1d-8a36-950f4532c235","Type":"ContainerDied","Data":"2a6695547cd8fb8655e22ecc4e0bd5805ae43435b8bf2e0db22838e7034dfc54"}
Oct 30 19:50:28.924765 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.924339    6030 scope.go:122] "RemoveContainer" containerID="0ce3f341a4f37975066408580f75e7b88b8392c74421c14a6d49838c42612260"
Oct 30 19:50:28.928291 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.928268    6030 generic.go:356] "Generic (PLEG): container finished" podID="74ede63b-4f53-4e61-ab60-4b79ccb3101d" containerID="a4bb907def79b77fd836d4ff86c2b650a37366a7957b9db18db2634f48af2c14" exitCode=0
Oct 30 19:50:28.929212 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.928345    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"
Oct 30 19:50:28.929212 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.928345    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf" event={"ID":"74ede63b-4f53-4e61-ab60-4b79ccb3101d","Type":"ContainerDied","Data":"a4bb907def79b77fd836d4ff86c2b650a37366a7957b9db18db2634f48af2c14"}
Oct 30 19:50:28.929212 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.928631    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf" event={"ID":"74ede63b-4f53-4e61-ab60-4b79ccb3101d","Type":"ContainerDied","Data":"bb5e5a644fe9dcd0efe1342e5a484a49cf1b99565a062817883e7d113052d9dc"}
Oct 30 19:50:28.945696 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.945283    6030 scope.go:122] "RemoveContainer" containerID="0ce3f341a4f37975066408580f75e7b88b8392c74421c14a6d49838c42612260"
Oct 30 19:50:28.948279 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:50:28.948242    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce3f341a4f37975066408580f75e7b88b8392c74421c14a6d49838c42612260\": container with ID starting with 0ce3f341a4f37975066408580f75e7b88b8392c74421c14a6d49838c42612260 not found: ID does not exist" containerID="0ce3f341a4f37975066408580f75e7b88b8392c74421c14a6d49838c42612260"
Oct 30 19:50:28.948414 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.948276    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce3f341a4f37975066408580f75e7b88b8392c74421c14a6d49838c42612260"} err="failed to get container status \"0ce3f341a4f37975066408580f75e7b88b8392c74421c14a6d49838c42612260\": rpc error: code = NotFound desc = could not find container \"0ce3f341a4f37975066408580f75e7b88b8392c74421c14a6d49838c42612260\": container with ID starting with 0ce3f341a4f37975066408580f75e7b88b8392c74421c14a6d49838c42612260 not found: ID does not exist"
Oct 30 19:50:28.948414 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.948296    6030 scope.go:122] "RemoveContainer" containerID="a4bb907def79b77fd836d4ff86c2b650a37366a7957b9db18db2634f48af2c14"
Oct 30 19:50:28.962328 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.962296    6030 scope.go:122] "RemoveContainer" containerID="a4bb907def79b77fd836d4ff86c2b650a37366a7957b9db18db2634f48af2c14"
Oct 30 19:50:28.962697 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:50:28.962672    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4bb907def79b77fd836d4ff86c2b650a37366a7957b9db18db2634f48af2c14\": container with ID starting with a4bb907def79b77fd836d4ff86c2b650a37366a7957b9db18db2634f48af2c14 not found: ID does not exist" containerID="a4bb907def79b77fd836d4ff86c2b650a37366a7957b9db18db2634f48af2c14"
Oct 30 19:50:28.962819 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.962711    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4bb907def79b77fd836d4ff86c2b650a37366a7957b9db18db2634f48af2c14"} err="failed to get container status \"a4bb907def79b77fd836d4ff86c2b650a37366a7957b9db18db2634f48af2c14\": rpc error: code = NotFound desc = could not find container \"a4bb907def79b77fd836d4ff86c2b650a37366a7957b9db18db2634f48af2c14\": container with ID starting with a4bb907def79b77fd836d4ff86c2b650a37366a7957b9db18db2634f48af2c14 not found: ID does not exist"
Oct 30 19:50:28.970612 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.970585    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"
Oct 30 19:50:28.974699 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.974671    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"]
Oct 30 19:50:28.976564 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:50:28.976290    6030 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ede63b_4f53_4e61_ab60_4b79ccb3101d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2d13b0_c920_4e1d_8a36_950f4532c235.slice/crio-2a6695547cd8fb8655e22ecc4e0bd5805ae43435b8bf2e0db22838e7034dfc54\": RecentStats: unable to find data in memory cache]"
Oct 30 19:50:28.984910 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.984886    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f2325-849994d776-qf6ph"]
Oct 30 19:50:29.001729 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:28.999608    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"]
Oct 30 19:50:29.009295 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.009264    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-9b997dc-d2jsf"]
Oct 30 19:50:29.048142 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.048103    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"]
Oct 30 19:50:29.054374 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.054342    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef436d3c-4p4wl"]
Oct 30 19:50:29.201513 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.201477    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a2d13b0-c920-4e1d-8a36-950f4532c235" path="/var/lib/kubelet/pods/1a2d13b0-c920-4e1d-8a36-950f4532c235/volumes"
Oct 30 19:50:29.201947 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.201928    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e221623-3b0d-4d13-b2e5-922a6ab8f9b4" path="/var/lib/kubelet/pods/5e221623-3b0d-4d13-b2e5-922a6ab8f9b4/volumes"
Oct 30 19:50:29.202381 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.202364    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ede63b-4f53-4e61-ab60-4b79ccb3101d" path="/var/lib/kubelet/pods/74ede63b-4f53-4e61-ab60-4b79ccb3101d/volumes"
Oct 30 19:50:29.231255 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.231229    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"]
Oct 30 19:50:29.231893 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:50:29.231864    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a2ed94_fdfe_41ac_aced_bc1f2c326d06.slice/crio-a0c9d244855a8a22c1ad69eee90278a517c04a8ba8272b72f8fd9b9eed96f78c WatchSource:0}: Error finding container a0c9d244855a8a22c1ad69eee90278a517c04a8ba8272b72f8fd9b9eed96f78c: Status 404 returned error can't find the container with id a0c9d244855a8a22c1ad69eee90278a517c04a8ba8272b72f8fd9b9eed96f78c
Oct 30 19:50:29.360550 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.360489    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"]
Oct 30 19:50:29.699957 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.699928    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:29.700399 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.700374    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:29.701057 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.701021    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:50:29.943283 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.943239    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q" event={"ID":"3919c09d-9f26-4233-a9b9-6f38d6e39970","Type":"ContainerStarted","Data":"d969184d93f51f77843f066b9ac8757c6e3f35e4a49ab6c1ff62f5e3b48bc1a5"}
Oct 30 19:50:29.943283 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.943288    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q" event={"ID":"3919c09d-9f26-4233-a9b9-6f38d6e39970","Type":"ContainerStarted","Data":"af62f4ec204e4b210f338cdeb336a17f4bc6c7f203c3e3cf8c812a33e0a80b86"}
Oct 30 19:50:29.943900 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.943379    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"
Oct 30 19:50:29.945124 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.945098    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf" event={"ID":"54a2ed94-fdfe-41ac-aced-bc1f2c326d06","Type":"ContainerStarted","Data":"238a06fcd487895a6dc880506b7a15412bf57da87beaadca253fc1ce57c536c4"}
Oct 30 19:50:29.945232 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.945146    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf" event={"ID":"54a2ed94-fdfe-41ac-aced-bc1f2c326d06","Type":"ContainerStarted","Data":"a0c9d244855a8a22c1ad69eee90278a517c04a8ba8272b72f8fd9b9eed96f78c"}
Oct 30 19:50:29.945272 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.945245    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"
Oct 30 19:50:29.966272 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.966170    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q" podStartSLOduration=1.9661523920000001 podStartE2EDuration="1.966152392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:29.964010877 +0000 UTC m=+1108.879617737" watchObservedRunningTime="2026-10-30 19:50:29.966152392 +0000 UTC m=+1108.881759250"
Oct 30 19:50:29.989000 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:29.988943    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf" podStartSLOduration=1.988923856 podStartE2EDuration="1.988923856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:29.986321447 +0000 UTC m=+1108.901928303" watchObservedRunningTime="2026-10-30 19:50:29.988923856 +0000 UTC m=+1108.904530719"
Oct 30 19:50:34.291850 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:50:34.291803    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:50:38.723370 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:38.723321    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"]
Oct 30 19:50:38.730743 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:38.730707    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"
Oct 30 19:50:38.736418 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:38.736391    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4f2325\""
Oct 30 19:50:38.738510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:38.738484    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"]
Oct 30 19:50:38.796401 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:38.796367    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlt7z\" (UniqueName: \"kubernetes.io/projected/1f131160-44fc-416c-9f1a-36786452d4db-kube-api-access-zlt7z\") pod \"payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc\" (UID: \"1f131160-44fc-416c-9f1a-36786452d4db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"
Oct 30 19:50:38.796580 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:38.796499    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f131160-44fc-416c-9f1a-36786452d4db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc\" (UID: \"1f131160-44fc-416c-9f1a-36786452d4db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"
Oct 30 19:50:38.901121 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:38.901074    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlt7z\" (UniqueName: \"kubernetes.io/projected/1f131160-44fc-416c-9f1a-36786452d4db-kube-api-access-zlt7z\") pod \"payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc\" (UID: \"1f131160-44fc-416c-9f1a-36786452d4db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"
Oct 30 19:50:38.916559 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:38.916522    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f131160-44fc-416c-9f1a-36786452d4db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc\" (UID: \"1f131160-44fc-416c-9f1a-36786452d4db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"
Oct 30 19:50:38.930129 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:38.917718    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f131160-44fc-416c-9f1a-36786452d4db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc\" (UID: \"1f131160-44fc-416c-9f1a-36786452d4db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"
Oct 30 19:50:38.989116 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:38.975776    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlt7z\" (UniqueName: \"kubernetes.io/projected/1f131160-44fc-416c-9f1a-36786452d4db-kube-api-access-zlt7z\") pod \"payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc\" (UID: \"1f131160-44fc-416c-9f1a-36786452d4db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"
Oct 30 19:50:39.052421 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:39.050799    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"
Oct 30 19:50:39.425006 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:39.424964    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"]
Oct 30 19:50:40.089628 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:40.089586    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc" event={"ID":"1f131160-44fc-416c-9f1a-36786452d4db","Type":"ContainerStarted","Data":"6ea841a96da1b6f665f981f9b44de70f17ea15e35e5be6e2742c204c3d285d29"}
Oct 30 19:50:40.090016 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:40.089654    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc" event={"ID":"1f131160-44fc-416c-9f1a-36786452d4db","Type":"ContainerStarted","Data":"bb34b8f74d38555d68152649d627a447464137b42421d46a6cc02c521c7583ad"}
Oct 30 19:50:40.090016 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:40.089814    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"
Oct 30 19:50:40.114758 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:40.114701    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc" podStartSLOduration=2.114687114 podStartE2EDuration="2.114687114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:40.114181658 +0000 UTC m=+1119.029788514" watchObservedRunningTime="2026-10-30 19:50:40.114687114 +0000 UTC m=+1119.030293969"
Oct 30 19:50:40.957287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:40.957260    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"
Oct 30 19:50:40.957990 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:40.957971    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"
Oct 30 19:50:42.032653 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:42.032617    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"]
Oct 30 19:50:42.038655 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:42.038631    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-346ed860-ntgg9"]
Oct 30 19:50:43.202010 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:43.201978    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca219c8b-7690-45c6-9ed9-dd04a5fd458c" path="/var/lib/kubelet/pods/ca219c8b-7690-45c6-9ed9-dd04a5fd458c/volumes"
Oct 30 19:50:45.677636 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:45.677593    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"]
Oct 30 19:50:45.698933 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:45.698898    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"]
Oct 30 19:50:50.691226 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:50.691144    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc" podUID="1f6f202f-9bf1-4e5e-b399-bbaf83e2adce" containerName="payload-processing" containerID="cri-o://5b931bddbffce81255c121c96737bb3f8d108f9f7f4837dfe3c933c5dbe4de77" gracePeriod=25
Oct 30 19:50:50.702210 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:50.702124    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx" podUID="a70f60f8-d58b-4db3-a200-4958351a5e47" containerName="payload-pre-processing" containerID="cri-o://d53f53c1bac89f524de63b59fa20c7029004113bfd34f61729c7ba262d3647d4" gracePeriod=25
Oct 30 19:50:51.045025 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.044997    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"
Oct 30 19:50:51.116166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.116030    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"
Oct 30 19:50:51.123693 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.123661    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"
Oct 30 19:50:51.198065 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.198020    6030 generic.go:356] "Generic (PLEG): container finished" podID="1f6f202f-9bf1-4e5e-b399-bbaf83e2adce" containerID="5b931bddbffce81255c121c96737bb3f8d108f9f7f4837dfe3c933c5dbe4de77" exitCode=0
Oct 30 19:50:51.198416 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.198392    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"
Oct 30 19:50:51.200903 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.200872    6030 generic.go:356] "Generic (PLEG): container finished" podID="a70f60f8-d58b-4db3-a200-4958351a5e47" containerID="d53f53c1bac89f524de63b59fa20c7029004113bfd34f61729c7ba262d3647d4" exitCode=0
Oct 30 19:50:51.201092 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.200985    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"
Oct 30 19:50:51.209135 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.209111    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc" event={"ID":"1f6f202f-9bf1-4e5e-b399-bbaf83e2adce","Type":"ContainerDied","Data":"5b931bddbffce81255c121c96737bb3f8d108f9f7f4837dfe3c933c5dbe4de77"}
Oct 30 19:50:51.209280 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.209166    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc" event={"ID":"1f6f202f-9bf1-4e5e-b399-bbaf83e2adce","Type":"ContainerDied","Data":"964e2a78cad4420c6bb1b402c69691034bb837cbff2dfb63f3625e488fe82b4b"}
Oct 30 19:50:51.209280 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.209183    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx" event={"ID":"a70f60f8-d58b-4db3-a200-4958351a5e47","Type":"ContainerDied","Data":"d53f53c1bac89f524de63b59fa20c7029004113bfd34f61729c7ba262d3647d4"}
Oct 30 19:50:51.209280 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.209206    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx" event={"ID":"a70f60f8-d58b-4db3-a200-4958351a5e47","Type":"ContainerDied","Data":"8befd2a16a12e7d868ab9d93b08878d914553b0ce363c3c0888203ba9c6c20da"}
Oct 30 19:50:51.209280 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.209225    6030 scope.go:122] "RemoveContainer" containerID="5b931bddbffce81255c121c96737bb3f8d108f9f7f4837dfe3c933c5dbe4de77"
Oct 30 19:50:51.210285 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.210264    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6f202f-9bf1-4e5e-b399-bbaf83e2adce-kube-api-access-zj5p8\" (UniqueName: \"kubernetes.io/projected/1f6f202f-9bf1-4e5e-b399-bbaf83e2adce-kube-api-access-zj5p8\") pod \"1f6f202f-9bf1-4e5e-b399-bbaf83e2adce\" (UID: \"1f6f202f-9bf1-4e5e-b399-bbaf83e2adce\") "
Oct 30 19:50:51.210547 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.210527    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f6f202f-9bf1-4e5e-b399-bbaf83e2adce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6f202f-9bf1-4e5e-b399-bbaf83e2adce-plugins-config-volume\") pod \"1f6f202f-9bf1-4e5e-b399-bbaf83e2adce\" (UID: \"1f6f202f-9bf1-4e5e-b399-bbaf83e2adce\") "
Oct 30 19:50:51.211213 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.211184    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6f202f-9bf1-4e5e-b399-bbaf83e2adce-plugins-config-volume" pod "1f6f202f-9bf1-4e5e-b399-bbaf83e2adce" (UID: "1f6f202f-9bf1-4e5e-b399-bbaf83e2adce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:50:51.212516 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.212495    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6f202f-9bf1-4e5e-b399-bbaf83e2adce-kube-api-access-zj5p8" pod "1f6f202f-9bf1-4e5e-b399-bbaf83e2adce" (UID: "1f6f202f-9bf1-4e5e-b399-bbaf83e2adce"). InnerVolumeSpecName "kube-api-access-zj5p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:51.221295 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.221269    6030 scope.go:122] "RemoveContainer" containerID="5b931bddbffce81255c121c96737bb3f8d108f9f7f4837dfe3c933c5dbe4de77"
Oct 30 19:50:51.221615 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:50:51.221591    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b931bddbffce81255c121c96737bb3f8d108f9f7f4837dfe3c933c5dbe4de77\": container with ID starting with 5b931bddbffce81255c121c96737bb3f8d108f9f7f4837dfe3c933c5dbe4de77 not found: ID does not exist" containerID="5b931bddbffce81255c121c96737bb3f8d108f9f7f4837dfe3c933c5dbe4de77"
Oct 30 19:50:51.221723 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.221624    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b931bddbffce81255c121c96737bb3f8d108f9f7f4837dfe3c933c5dbe4de77"} err="failed to get container status \"5b931bddbffce81255c121c96737bb3f8d108f9f7f4837dfe3c933c5dbe4de77\": rpc error: code = NotFound desc = could not find container \"5b931bddbffce81255c121c96737bb3f8d108f9f7f4837dfe3c933c5dbe4de77\": container with ID starting with 5b931bddbffce81255c121c96737bb3f8d108f9f7f4837dfe3c933c5dbe4de77 not found: ID does not exist"
Oct 30 19:50:51.221723 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.221642    6030 scope.go:122] "RemoveContainer" containerID="d53f53c1bac89f524de63b59fa20c7029004113bfd34f61729c7ba262d3647d4"
Oct 30 19:50:51.230961 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.230940    6030 scope.go:122] "RemoveContainer" containerID="d53f53c1bac89f524de63b59fa20c7029004113bfd34f61729c7ba262d3647d4"
Oct 30 19:50:51.231294 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:50:51.231275    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d53f53c1bac89f524de63b59fa20c7029004113bfd34f61729c7ba262d3647d4\": container with ID starting with d53f53c1bac89f524de63b59fa20c7029004113bfd34f61729c7ba262d3647d4 not found: ID does not exist" containerID="d53f53c1bac89f524de63b59fa20c7029004113bfd34f61729c7ba262d3647d4"
Oct 30 19:50:51.231363 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.231303    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d53f53c1bac89f524de63b59fa20c7029004113bfd34f61729c7ba262d3647d4"} err="failed to get container status \"d53f53c1bac89f524de63b59fa20c7029004113bfd34f61729c7ba262d3647d4\": rpc error: code = NotFound desc = could not find container \"d53f53c1bac89f524de63b59fa20c7029004113bfd34f61729c7ba262d3647d4\": container with ID starting with d53f53c1bac89f524de63b59fa20c7029004113bfd34f61729c7ba262d3647d4 not found: ID does not exist"
Oct 30 19:50:51.312064 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.311972    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70f60f8-d58b-4db3-a200-4958351a5e47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70f60f8-d58b-4db3-a200-4958351a5e47-plugins-config-volume\") pod \"a70f60f8-d58b-4db3-a200-4958351a5e47\" (UID: \"a70f60f8-d58b-4db3-a200-4958351a5e47\") "
Oct 30 19:50:51.312238 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.312065    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70f60f8-d58b-4db3-a200-4958351a5e47-kube-api-access-kzsqr\" (UniqueName: \"kubernetes.io/projected/a70f60f8-d58b-4db3-a200-4958351a5e47-kube-api-access-kzsqr\") pod \"a70f60f8-d58b-4db3-a200-4958351a5e47\" (UID: \"a70f60f8-d58b-4db3-a200-4958351a5e47\") "
Oct 30 19:50:51.312844 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.312376    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70f60f8-d58b-4db3-a200-4958351a5e47-plugins-config-volume" pod "a70f60f8-d58b-4db3-a200-4958351a5e47" (UID: "a70f60f8-d58b-4db3-a200-4958351a5e47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:50:51.312844 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.312597    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj5p8\" (UniqueName: \"kubernetes.io/projected/1f6f202f-9bf1-4e5e-b399-bbaf83e2adce-kube-api-access-zj5p8\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:51.312844 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.312615    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70f60f8-d58b-4db3-a200-4958351a5e47-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:51.312844 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.312630    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6f202f-9bf1-4e5e-b399-bbaf83e2adce-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:51.314414 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.314392    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70f60f8-d58b-4db3-a200-4958351a5e47-kube-api-access-kzsqr" pod "a70f60f8-d58b-4db3-a200-4958351a5e47" (UID: "a70f60f8-d58b-4db3-a200-4958351a5e47"). InnerVolumeSpecName "kube-api-access-kzsqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:51.413575 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.413540    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzsqr\" (UniqueName: \"kubernetes.io/projected/a70f60f8-d58b-4db3-a200-4958351a5e47-kube-api-access-kzsqr\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:51.531597 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.531559    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"]
Oct 30 19:50:51.538486 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.538463    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1e3723-5b496fc79f-r2slx"]
Oct 30 19:50:51.554503 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.554473    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"]
Oct 30 19:50:51.560458 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:51.560437    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1e3723-7664ccccb4-tl8pc"]
Oct 30 19:50:53.202143 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:53.202110    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f6f202f-9bf1-4e5e-b399-bbaf83e2adce" path="/var/lib/kubelet/pods/1f6f202f-9bf1-4e5e-b399-bbaf83e2adce/volumes"
Oct 30 19:50:53.202558 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:50:53.202544    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a70f60f8-d58b-4db3-a200-4958351a5e47" path="/var/lib/kubelet/pods/a70f60f8-d58b-4db3-a200-4958351a5e47/volumes"
Oct 30 19:51:01.192076 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.191581    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"]
Oct 30 19:51:01.196733 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.192791    6030 cpu_manager.go:414] "containerMap: removing container" podUID="1f6f202f-9bf1-4e5e-b399-bbaf83e2adce" containerName="payload-processing"
Oct 30 19:51:01.196733 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.192828    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f6f202f-9bf1-4e5e-b399-bbaf83e2adce" containerName="payload-processing"
Oct 30 19:51:01.196733 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.192843    6030 cpu_manager.go:414] "containerMap: removing container" podUID="a70f60f8-d58b-4db3-a200-4958351a5e47" containerName="payload-pre-processing"
Oct 30 19:51:01.196733 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.192851    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70f60f8-d58b-4db3-a200-4958351a5e47" containerName="payload-pre-processing"
Oct 30 19:51:01.196733 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.193067    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f6f202f-9bf1-4e5e-b399-bbaf83e2adce" containerName="payload-processing"
Oct 30 19:51:01.196733 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.193083    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70f60f8-d58b-4db3-a200-4958351a5e47" containerName="payload-pre-processing"
Oct 30 19:51:01.407406 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.313628    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"
Oct 30 19:51:01.407406 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.349974    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxdms\" (UniqueName: \"kubernetes.io/projected/857bb812-0f79-433c-8f90-8a2a57232114-kube-api-access-dxdms\") pod \"payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4\" (UID: \"857bb812-0f79-433c-8f90-8a2a57232114\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"
Oct 30 19:51:01.407406 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.400537    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/857bb812-0f79-433c-8f90-8a2a57232114-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4\" (UID: \"857bb812-0f79-433c-8f90-8a2a57232114\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"
Oct 30 19:51:01.407406 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.392737    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"]
Oct 30 19:51:01.415724 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.415693    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ec5e0f-dockercfg-mptzf\""
Oct 30 19:51:01.416229 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.416118    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ec5e0f\""
Oct 30 19:51:01.503233 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.502935    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/857bb812-0f79-433c-8f90-8a2a57232114-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4\" (UID: \"857bb812-0f79-433c-8f90-8a2a57232114\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"
Oct 30 19:51:01.503233 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.503188    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxdms\" (UniqueName: \"kubernetes.io/projected/857bb812-0f79-433c-8f90-8a2a57232114-kube-api-access-dxdms\") pod \"payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4\" (UID: \"857bb812-0f79-433c-8f90-8a2a57232114\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"
Oct 30 19:51:01.504497 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.504457    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/857bb812-0f79-433c-8f90-8a2a57232114-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4\" (UID: \"857bb812-0f79-433c-8f90-8a2a57232114\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"
Oct 30 19:51:01.558507 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.557911    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxdms\" (UniqueName: \"kubernetes.io/projected/857bb812-0f79-433c-8f90-8a2a57232114-kube-api-access-dxdms\") pod \"payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4\" (UID: \"857bb812-0f79-433c-8f90-8a2a57232114\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"
Oct 30 19:51:01.723006 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.722969    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"
Oct 30 19:51:01.855280 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.855243    6030 scope.go:122] "RemoveContainer" containerID="7da3188965c5419129016e5b6865f6a6619d471b731d7156edc7504a3d8ee946"
Oct 30 19:51:01.866719 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.866698    6030 scope.go:122] "RemoveContainer" containerID="0191d5214230e0e1ba54be6ec1f06692faad93b0735363fc6ea6c039fc49b71c"
Oct 30 19:51:01.876053 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:01.876019    6030 scope.go:122] "RemoveContainer" containerID="1eb8494f85ead8143d271061b9c1af5d585889cd25be4091f5eb66c29e7d8fb6"
Oct 30 19:51:02.016879 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:02.016851    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"]
Oct 30 19:51:02.017562 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:51:02.017537    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857bb812_0f79_433c_8f90_8a2a57232114.slice/crio-7d734aeab6410a6a8e01dfcf4cd39935996ee9e4b4abddbf6ccac3dcb7cd4f62 WatchSource:0}: Error finding container 7d734aeab6410a6a8e01dfcf4cd39935996ee9e4b4abddbf6ccac3dcb7cd4f62: Status 404 returned error can't find the container with id 7d734aeab6410a6a8e01dfcf4cd39935996ee9e4b4abddbf6ccac3dcb7cd4f62
Oct 30 19:51:02.444456 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:02.444418    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4" event={"ID":"857bb812-0f79-433c-8f90-8a2a57232114","Type":"ContainerStarted","Data":"9bde9fd0299ccac42a7e01887694117089735053f4dd5a17aa12326937ae0ac3"}
Oct 30 19:51:02.450150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:02.450103    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4" event={"ID":"857bb812-0f79-433c-8f90-8a2a57232114","Type":"ContainerStarted","Data":"7d734aeab6410a6a8e01dfcf4cd39935996ee9e4b4abddbf6ccac3dcb7cd4f62"}
Oct 30 19:51:02.451916 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:02.451885    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"
Oct 30 19:51:02.488477 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:02.488417    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4" podStartSLOduration=1.488397184 podStartE2EDuration="1.488397184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:02.479575849 +0000 UTC m=+1141.395182706" watchObservedRunningTime="2026-10-30 19:51:02.488397184 +0000 UTC m=+1141.404004040"
Oct 30 19:51:04.292570 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:51:04.292536    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:51:11.494639 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:11.494602    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"]
Oct 30 19:51:11.495316 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:11.494899    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m" podUID="39e3cf00-165f-4cd0-b98e-4713a8ffbda1" containerName="maas-api" containerID="cri-o://14a03f89046bdc855709d18502a6dd610297771136a5dac82c3bc5e3adc3619d" gracePeriod=30
Oct 30 19:51:12.045107 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.045039    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:51:12.166308 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.166279    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-openshift-service-ca\") pod \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\" (UID: \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\") "
Oct 30 19:51:12.166482 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.166406    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-maas-api-tls\") pod \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\" (UID: \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\") "
Oct 30 19:51:12.166482 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.166468    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-kube-api-access-9vlk2\" (UniqueName: \"kubernetes.io/projected/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-kube-api-access-9vlk2\") pod \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\" (UID: \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\") "
Oct 30 19:51:12.166569 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.166551    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-metrics-certs\") pod \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\" (UID: \"39e3cf00-165f-4cd0-b98e-4713a8ffbda1\") "
Oct 30 19:51:12.166679 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.166652    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-openshift-service-ca" pod "39e3cf00-165f-4cd0-b98e-4713a8ffbda1" (UID: "39e3cf00-165f-4cd0-b98e-4713a8ffbda1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:12.167100 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.167073    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:12.168750 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.168728    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-maas-api-tls" pod "39e3cf00-165f-4cd0-b98e-4713a8ffbda1" (UID: "39e3cf00-165f-4cd0-b98e-4713a8ffbda1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:12.169285 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.169268    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-kube-api-access-9vlk2" pod "39e3cf00-165f-4cd0-b98e-4713a8ffbda1" (UID: "39e3cf00-165f-4cd0-b98e-4713a8ffbda1"). InnerVolumeSpecName "kube-api-access-9vlk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:12.169285 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.169265    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-metrics-certs" pod "39e3cf00-165f-4cd0-b98e-4713a8ffbda1" (UID: "39e3cf00-165f-4cd0-b98e-4713a8ffbda1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:12.268510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.268472    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:12.268510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.268509    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vlk2\" (UniqueName: \"kubernetes.io/projected/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-kube-api-access-9vlk2\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:12.268716 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.268525    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e3cf00-165f-4cd0-b98e-4713a8ffbda1-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:12.532236 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.532202    6030 generic.go:356] "Generic (PLEG): container finished" podID="39e3cf00-165f-4cd0-b98e-4713a8ffbda1" containerID="14a03f89046bdc855709d18502a6dd610297771136a5dac82c3bc5e3adc3619d" exitCode=0
Oct 30 19:51:12.532721 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.532248    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"
Oct 30 19:51:12.532721 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.532252    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m" event={"ID":"39e3cf00-165f-4cd0-b98e-4713a8ffbda1","Type":"ContainerDied","Data":"14a03f89046bdc855709d18502a6dd610297771136a5dac82c3bc5e3adc3619d"}
Oct 30 19:51:12.532721 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.532311    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m" event={"ID":"39e3cf00-165f-4cd0-b98e-4713a8ffbda1","Type":"ContainerDied","Data":"47a7509335fea8732c10fac9dfa4fb4568893fa758f558ab628a02b9c3910f35"}
Oct 30 19:51:12.532721 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.532342    6030 scope.go:122] "RemoveContainer" containerID="14a03f89046bdc855709d18502a6dd610297771136a5dac82c3bc5e3adc3619d"
Oct 30 19:51:12.546140 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.546110    6030 scope.go:122] "RemoveContainer" containerID="14a03f89046bdc855709d18502a6dd610297771136a5dac82c3bc5e3adc3619d"
Oct 30 19:51:12.547027 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:51:12.546998    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a03f89046bdc855709d18502a6dd610297771136a5dac82c3bc5e3adc3619d\": container with ID starting with 14a03f89046bdc855709d18502a6dd610297771136a5dac82c3bc5e3adc3619d not found: ID does not exist" containerID="14a03f89046bdc855709d18502a6dd610297771136a5dac82c3bc5e3adc3619d"
Oct 30 19:51:12.551264 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.547075    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a03f89046bdc855709d18502a6dd610297771136a5dac82c3bc5e3adc3619d"} err="failed to get container status \"14a03f89046bdc855709d18502a6dd610297771136a5dac82c3bc5e3adc3619d\": rpc error: code = NotFound desc = could not find container \"14a03f89046bdc855709d18502a6dd610297771136a5dac82c3bc5e3adc3619d\": container with ID starting with 14a03f89046bdc855709d18502a6dd610297771136a5dac82c3bc5e3adc3619d not found: ID does not exist"
Oct 30 19:51:12.562377 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.562345    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"]
Oct 30 19:51:12.565178 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:12.565155    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1e3723-7d5cb84cdf-pz94m"]
Oct 30 19:51:13.201755 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:13.201721    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e3cf00-165f-4cd0-b98e-4713a8ffbda1" path="/var/lib/kubelet/pods/39e3cf00-165f-4cd0-b98e-4713a8ffbda1/volumes"
Oct 30 19:51:14.460552 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:14.460523    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"
Oct 30 19:51:16.300119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:16.300085    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"]
Oct 30 19:51:16.601124 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:16.596721    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx" podUID="45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" containerName="istio-proxy" containerID="cri-o://36555afb27906c5993066b65d501ccab0914430b7c230476c4a49e15c1c851d9" gracePeriod=30
Oct 30 19:51:22.327309 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.327284    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:51:22.424487 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.424398    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-workload-socket\") pod \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") "
Oct 30 19:51:22.424487 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.424459    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istiod-ca-cert\") pod \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") "
Oct 30 19:51:22.424726 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.424658    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-data\") pod \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") "
Oct 30 19:51:22.424726 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.424690    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-workload-certs\") pod \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") "
Oct 30 19:51:22.424726 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.424693    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-workload-socket" pod "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" (UID: "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:22.424893 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.424735    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-kube-api-access-58zwg\" (UniqueName: \"kubernetes.io/projected/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-kube-api-access-58zwg\") pod \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") "
Oct 30 19:51:22.424893 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.424768    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-token\" (UniqueName: \"kubernetes.io/projected/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-token\") pod \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") "
Oct 30 19:51:22.424893 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.424845    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-podinfo\") pod \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") "
Oct 30 19:51:22.424893 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.424873    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-credential-socket\") pod \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") "
Oct 30 19:51:22.425189 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.424899    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-data" pod "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" (UID: "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:22.425189 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.424933    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-envoy\") pod \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\" (UID: \"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8\") "
Oct 30 19:51:22.425189 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.424938    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istiod-ca-cert" pod "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" (UID: "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:22.425189 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.424961    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-workload-certs" pod "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" (UID: "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:22.425189 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.425148    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-credential-socket" pod "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" (UID: "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:22.425472 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.425450    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:22.425472 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.425472    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:22.425654 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.425486    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:22.425654 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.425500    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:22.425654 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.425512    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:22.427144 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.427120    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-podinfo" pod "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" (UID: "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:51:22.427325 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.427304    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-envoy" pod "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" (UID: "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:22.427531 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.427512    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-kube-api-access-58zwg" pod "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" (UID: "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8"). InnerVolumeSpecName "kube-api-access-58zwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:22.427596 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.427524    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-token" pod "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" (UID: "45eef3eb-e56b-4e28-ae5c-02b5eeae51b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:22.526639 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.526604    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58zwg\" (UniqueName: \"kubernetes.io/projected/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-kube-api-access-58zwg\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:22.526639 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.526632    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:22.526639 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.526643    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:22.526883 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.526652    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:22.679619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.679518    6030 generic.go:356] "Generic (PLEG): container finished" podID="45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" containerID="36555afb27906c5993066b65d501ccab0914430b7c230476c4a49e15c1c851d9" exitCode=0
Oct 30 19:51:22.679619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.679567    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx" event={"ID":"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8","Type":"ContainerDied","Data":"36555afb27906c5993066b65d501ccab0914430b7c230476c4a49e15c1c851d9"}
Oct 30 19:51:22.679619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.679613    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx" event={"ID":"45eef3eb-e56b-4e28-ae5c-02b5eeae51b8","Type":"ContainerDied","Data":"761d1a89c1a25035894b25f30a7a0b78f8b2374d384c200006f4789ab84747ce"}
Oct 30 19:51:22.679871 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.679634    6030 scope.go:122] "RemoveContainer" containerID="36555afb27906c5993066b65d501ccab0914430b7c230476c4a49e15c1c851d9"
Oct 30 19:51:22.679871 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.679571    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"
Oct 30 19:51:22.691159 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.691142    6030 scope.go:122] "RemoveContainer" containerID="2d85ef91684daaf90ad19f582411a2b74d4ff3f815567cabc3f0a3d0877d5a5b"
Oct 30 19:51:22.701169 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.701149    6030 scope.go:122] "RemoveContainer" containerID="36555afb27906c5993066b65d501ccab0914430b7c230476c4a49e15c1c851d9"
Oct 30 19:51:22.701447 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:51:22.701430    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36555afb27906c5993066b65d501ccab0914430b7c230476c4a49e15c1c851d9\": container with ID starting with 36555afb27906c5993066b65d501ccab0914430b7c230476c4a49e15c1c851d9 not found: ID does not exist" containerID="36555afb27906c5993066b65d501ccab0914430b7c230476c4a49e15c1c851d9"
Oct 30 19:51:22.701507 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.701457    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36555afb27906c5993066b65d501ccab0914430b7c230476c4a49e15c1c851d9"} err="failed to get container status \"36555afb27906c5993066b65d501ccab0914430b7c230476c4a49e15c1c851d9\": rpc error: code = NotFound desc = could not find container \"36555afb27906c5993066b65d501ccab0914430b7c230476c4a49e15c1c851d9\": container with ID starting with 36555afb27906c5993066b65d501ccab0914430b7c230476c4a49e15c1c851d9 not found: ID does not exist"
Oct 30 19:51:22.701507 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.701475    6030 scope.go:122] "RemoveContainer" containerID="2d85ef91684daaf90ad19f582411a2b74d4ff3f815567cabc3f0a3d0877d5a5b"
Oct 30 19:51:22.701685 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:51:22.701667    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d85ef91684daaf90ad19f582411a2b74d4ff3f815567cabc3f0a3d0877d5a5b\": container with ID starting with 2d85ef91684daaf90ad19f582411a2b74d4ff3f815567cabc3f0a3d0877d5a5b not found: ID does not exist" containerID="2d85ef91684daaf90ad19f582411a2b74d4ff3f815567cabc3f0a3d0877d5a5b"
Oct 30 19:51:22.701728 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.701692    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d85ef91684daaf90ad19f582411a2b74d4ff3f815567cabc3f0a3d0877d5a5b"} err="failed to get container status \"2d85ef91684daaf90ad19f582411a2b74d4ff3f815567cabc3f0a3d0877d5a5b\": rpc error: code = NotFound desc = could not find container \"2d85ef91684daaf90ad19f582411a2b74d4ff3f815567cabc3f0a3d0877d5a5b\": container with ID starting with 2d85ef91684daaf90ad19f582411a2b74d4ff3f815567cabc3f0a3d0877d5a5b not found: ID does not exist"
Oct 30 19:51:22.711442 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.711420    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"]
Oct 30 19:51:22.715905 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:22.715884    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c9d69c-openshift-default-7cf99b5679-x8svx"]
Oct 30 19:51:23.201784 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:23.201754    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" path="/var/lib/kubelet/pods/45eef3eb-e56b-4e28-ae5c-02b5eeae51b8/volumes"
Oct 30 19:51:30.689845 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:30.689797    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"]
Oct 30 19:51:30.690352 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:30.690094    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk" podUID="c1feae8e-f123-45b9-a52b-86457fabe1ec" containerName="maas-api" containerID="cri-o://2c42a7215e63d71eadf7a04766a545c544f8607fcd87010e25d60da155ee9442" gracePeriod=30
Oct 30 19:51:30.772250 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:30.772217    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"]
Oct 30 19:51:30.772514 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:30.772464    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q" podUID="3919c09d-9f26-4233-a9b9-6f38d6e39970" containerName="payload-processing" containerID="cri-o://d969184d93f51f77843f066b9ac8757c6e3f35e4a49ab6c1ff62f5e3b48bc1a5" gracePeriod=30
Oct 30 19:51:30.781709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:30.781647    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"]
Oct 30 19:51:30.781841 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:30.781821    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf" podUID="54a2ed94-fdfe-41ac-aced-bc1f2c326d06" containerName="payload-pre-processing" containerID="cri-o://238a06fcd487895a6dc880506b7a15412bf57da87beaadca253fc1ce57c536c4" gracePeriod=30
Oct 30 19:51:30.955470 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:30.955367    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf" podUID="54a2ed94-fdfe-41ac-aced-bc1f2c326d06" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.104:9004: connect: connection refused"
Oct 30 19:51:30.957252 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:30.957201    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q" podUID="3919c09d-9f26-4233-a9b9-6f38d6e39970" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.105:9004: connect: connection refused"
Oct 30 19:51:31.395101 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.382885    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"
Oct 30 19:51:31.446957 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.446930    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:51:31.451342 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.451317    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"
Oct 30 19:51:31.469880 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.469845    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3919c09d-9f26-4233-a9b9-6f38d6e39970-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3919c09d-9f26-4233-a9b9-6f38d6e39970-plugins-config-volume\") pod \"3919c09d-9f26-4233-a9b9-6f38d6e39970\" (UID: \"3919c09d-9f26-4233-a9b9-6f38d6e39970\") "
Oct 30 19:51:31.470096 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.469904    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1feae8e-f123-45b9-a52b-86457fabe1ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1feae8e-f123-45b9-a52b-86457fabe1ec-openshift-service-ca\") pod \"c1feae8e-f123-45b9-a52b-86457fabe1ec\" (UID: \"c1feae8e-f123-45b9-a52b-86457fabe1ec\") "
Oct 30 19:51:31.470096 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.470015    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1feae8e-f123-45b9-a52b-86457fabe1ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1feae8e-f123-45b9-a52b-86457fabe1ec-maas-api-tls\") pod \"c1feae8e-f123-45b9-a52b-86457fabe1ec\" (UID: \"c1feae8e-f123-45b9-a52b-86457fabe1ec\") "
Oct 30 19:51:31.470252 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.470229    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a2ed94-fdfe-41ac-aced-bc1f2c326d06-kube-api-access-nr65n\" (UniqueName: \"kubernetes.io/projected/54a2ed94-fdfe-41ac-aced-bc1f2c326d06-kube-api-access-nr65n\") pod \"54a2ed94-fdfe-41ac-aced-bc1f2c326d06\" (UID: \"54a2ed94-fdfe-41ac-aced-bc1f2c326d06\") "
Oct 30 19:51:31.470571 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.470315    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3919c09d-9f26-4233-a9b9-6f38d6e39970-kube-api-access-2l7cv\" (UniqueName: \"kubernetes.io/projected/3919c09d-9f26-4233-a9b9-6f38d6e39970-kube-api-access-2l7cv\") pod \"3919c09d-9f26-4233-a9b9-6f38d6e39970\" (UID: \"3919c09d-9f26-4233-a9b9-6f38d6e39970\") "
Oct 30 19:51:31.470571 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.470338    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3919c09d-9f26-4233-a9b9-6f38d6e39970-plugins-config-volume" pod "3919c09d-9f26-4233-a9b9-6f38d6e39970" (UID: "3919c09d-9f26-4233-a9b9-6f38d6e39970"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:31.470571 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.470375    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1feae8e-f123-45b9-a52b-86457fabe1ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1feae8e-f123-45b9-a52b-86457fabe1ec-metrics-certs\") pod \"c1feae8e-f123-45b9-a52b-86457fabe1ec\" (UID: \"c1feae8e-f123-45b9-a52b-86457fabe1ec\") "
Oct 30 19:51:31.470571 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.470413    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a2ed94-fdfe-41ac-aced-bc1f2c326d06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a2ed94-fdfe-41ac-aced-bc1f2c326d06-plugins-config-volume\") pod \"54a2ed94-fdfe-41ac-aced-bc1f2c326d06\" (UID: \"54a2ed94-fdfe-41ac-aced-bc1f2c326d06\") "
Oct 30 19:51:31.470571 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.470464    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1feae8e-f123-45b9-a52b-86457fabe1ec-kube-api-access-g7wmp\" (UniqueName: \"kubernetes.io/projected/c1feae8e-f123-45b9-a52b-86457fabe1ec-kube-api-access-g7wmp\") pod \"c1feae8e-f123-45b9-a52b-86457fabe1ec\" (UID: \"c1feae8e-f123-45b9-a52b-86457fabe1ec\") "
Oct 30 19:51:31.471158 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.471136    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3919c09d-9f26-4233-a9b9-6f38d6e39970-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:31.471490 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.471430    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a2ed94-fdfe-41ac-aced-bc1f2c326d06-plugins-config-volume" pod "54a2ed94-fdfe-41ac-aced-bc1f2c326d06" (UID: "54a2ed94-fdfe-41ac-aced-bc1f2c326d06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:31.471607 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.471518    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1feae8e-f123-45b9-a52b-86457fabe1ec-openshift-service-ca" pod "c1feae8e-f123-45b9-a52b-86457fabe1ec" (UID: "c1feae8e-f123-45b9-a52b-86457fabe1ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:31.473878 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.473853    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1feae8e-f123-45b9-a52b-86457fabe1ec-kube-api-access-g7wmp" pod "c1feae8e-f123-45b9-a52b-86457fabe1ec" (UID: "c1feae8e-f123-45b9-a52b-86457fabe1ec"). InnerVolumeSpecName "kube-api-access-g7wmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:31.474970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.474616    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1feae8e-f123-45b9-a52b-86457fabe1ec-maas-api-tls" pod "c1feae8e-f123-45b9-a52b-86457fabe1ec" (UID: "c1feae8e-f123-45b9-a52b-86457fabe1ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:31.474970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.474648    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1feae8e-f123-45b9-a52b-86457fabe1ec-metrics-certs" pod "c1feae8e-f123-45b9-a52b-86457fabe1ec" (UID: "c1feae8e-f123-45b9-a52b-86457fabe1ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:31.474970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.474740    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3919c09d-9f26-4233-a9b9-6f38d6e39970-kube-api-access-2l7cv" pod "3919c09d-9f26-4233-a9b9-6f38d6e39970" (UID: "3919c09d-9f26-4233-a9b9-6f38d6e39970"). InnerVolumeSpecName "kube-api-access-2l7cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:31.477122 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.477098    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a2ed94-fdfe-41ac-aced-bc1f2c326d06-kube-api-access-nr65n" pod "54a2ed94-fdfe-41ac-aced-bc1f2c326d06" (UID: "54a2ed94-fdfe-41ac-aced-bc1f2c326d06"). InnerVolumeSpecName "kube-api-access-nr65n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:31.573176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.573083    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1feae8e-f123-45b9-a52b-86457fabe1ec-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:31.573176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.573114    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr65n\" (UniqueName: \"kubernetes.io/projected/54a2ed94-fdfe-41ac-aced-bc1f2c326d06-kube-api-access-nr65n\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:31.573176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.573127    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l7cv\" (UniqueName: \"kubernetes.io/projected/3919c09d-9f26-4233-a9b9-6f38d6e39970-kube-api-access-2l7cv\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:31.573176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.573136    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1feae8e-f123-45b9-a52b-86457fabe1ec-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:31.573176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.573144    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a2ed94-fdfe-41ac-aced-bc1f2c326d06-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:31.573176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.573154    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7wmp\" (UniqueName: \"kubernetes.io/projected/c1feae8e-f123-45b9-a52b-86457fabe1ec-kube-api-access-g7wmp\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:31.573176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.573164    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1feae8e-f123-45b9-a52b-86457fabe1ec-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:31.775140 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.775102    6030 generic.go:356] "Generic (PLEG): container finished" podID="c1feae8e-f123-45b9-a52b-86457fabe1ec" containerID="2c42a7215e63d71eadf7a04766a545c544f8607fcd87010e25d60da155ee9442" exitCode=0
Oct 30 19:51:31.775565 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.775161    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"
Oct 30 19:51:31.775565 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.775174    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk" event={"ID":"c1feae8e-f123-45b9-a52b-86457fabe1ec","Type":"ContainerDied","Data":"2c42a7215e63d71eadf7a04766a545c544f8607fcd87010e25d60da155ee9442"}
Oct 30 19:51:31.775565 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.775232    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk" event={"ID":"c1feae8e-f123-45b9-a52b-86457fabe1ec","Type":"ContainerDied","Data":"dd434ced30b06f8b64fdb9e649e1d277a6ede480410674c0fa74cf015da37572"}
Oct 30 19:51:31.775565 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.775255    6030 scope.go:122] "RemoveContainer" containerID="2c42a7215e63d71eadf7a04766a545c544f8607fcd87010e25d60da155ee9442"
Oct 30 19:51:31.777087 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.777030    6030 generic.go:356] "Generic (PLEG): container finished" podID="3919c09d-9f26-4233-a9b9-6f38d6e39970" containerID="d969184d93f51f77843f066b9ac8757c6e3f35e4a49ab6c1ff62f5e3b48bc1a5" exitCode=0
Oct 30 19:51:31.777087 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.777076    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q" event={"ID":"3919c09d-9f26-4233-a9b9-6f38d6e39970","Type":"ContainerDied","Data":"d969184d93f51f77843f066b9ac8757c6e3f35e4a49ab6c1ff62f5e3b48bc1a5"}
Oct 30 19:51:31.777274 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.777092    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"
Oct 30 19:51:31.777274 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.777127    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q" event={"ID":"3919c09d-9f26-4233-a9b9-6f38d6e39970","Type":"ContainerDied","Data":"af62f4ec204e4b210f338cdeb336a17f4bc6c7f203c3e3cf8c812a33e0a80b86"}
Oct 30 19:51:31.778718 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.778696    6030 generic.go:356] "Generic (PLEG): container finished" podID="54a2ed94-fdfe-41ac-aced-bc1f2c326d06" containerID="238a06fcd487895a6dc880506b7a15412bf57da87beaadca253fc1ce57c536c4" exitCode=0
Oct 30 19:51:31.778839 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.778783    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"
Oct 30 19:51:31.778839 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.778785    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf" event={"ID":"54a2ed94-fdfe-41ac-aced-bc1f2c326d06","Type":"ContainerDied","Data":"238a06fcd487895a6dc880506b7a15412bf57da87beaadca253fc1ce57c536c4"}
Oct 30 19:51:31.778839 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.778835    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf" event={"ID":"54a2ed94-fdfe-41ac-aced-bc1f2c326d06","Type":"ContainerDied","Data":"a0c9d244855a8a22c1ad69eee90278a517c04a8ba8272b72f8fd9b9eed96f78c"}
Oct 30 19:51:31.787798 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.787678    6030 scope.go:122] "RemoveContainer" containerID="2c42a7215e63d71eadf7a04766a545c544f8607fcd87010e25d60da155ee9442"
Oct 30 19:51:31.788086 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:51:31.788021    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c42a7215e63d71eadf7a04766a545c544f8607fcd87010e25d60da155ee9442\": container with ID starting with 2c42a7215e63d71eadf7a04766a545c544f8607fcd87010e25d60da155ee9442 not found: ID does not exist" containerID="2c42a7215e63d71eadf7a04766a545c544f8607fcd87010e25d60da155ee9442"
Oct 30 19:51:31.788193 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.788095    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c42a7215e63d71eadf7a04766a545c544f8607fcd87010e25d60da155ee9442"} err="failed to get container status \"2c42a7215e63d71eadf7a04766a545c544f8607fcd87010e25d60da155ee9442\": rpc error: code = NotFound desc = could not find container \"2c42a7215e63d71eadf7a04766a545c544f8607fcd87010e25d60da155ee9442\": container with ID starting with 2c42a7215e63d71eadf7a04766a545c544f8607fcd87010e25d60da155ee9442 not found: ID does not exist"
Oct 30 19:51:31.788193 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.788120    6030 scope.go:122] "RemoveContainer" containerID="d969184d93f51f77843f066b9ac8757c6e3f35e4a49ab6c1ff62f5e3b48bc1a5"
Oct 30 19:51:31.797562 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.797544    6030 scope.go:122] "RemoveContainer" containerID="d969184d93f51f77843f066b9ac8757c6e3f35e4a49ab6c1ff62f5e3b48bc1a5"
Oct 30 19:51:31.797856 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:51:31.797835    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d969184d93f51f77843f066b9ac8757c6e3f35e4a49ab6c1ff62f5e3b48bc1a5\": container with ID starting with d969184d93f51f77843f066b9ac8757c6e3f35e4a49ab6c1ff62f5e3b48bc1a5 not found: ID does not exist" containerID="d969184d93f51f77843f066b9ac8757c6e3f35e4a49ab6c1ff62f5e3b48bc1a5"
Oct 30 19:51:31.797943 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.797865    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d969184d93f51f77843f066b9ac8757c6e3f35e4a49ab6c1ff62f5e3b48bc1a5"} err="failed to get container status \"d969184d93f51f77843f066b9ac8757c6e3f35e4a49ab6c1ff62f5e3b48bc1a5\": rpc error: code = NotFound desc = could not find container \"d969184d93f51f77843f066b9ac8757c6e3f35e4a49ab6c1ff62f5e3b48bc1a5\": container with ID starting with d969184d93f51f77843f066b9ac8757c6e3f35e4a49ab6c1ff62f5e3b48bc1a5 not found: ID does not exist"
Oct 30 19:51:31.797943 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.797884    6030 scope.go:122] "RemoveContainer" containerID="238a06fcd487895a6dc880506b7a15412bf57da87beaadca253fc1ce57c536c4"
Oct 30 19:51:31.806571 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.806553    6030 scope.go:122] "RemoveContainer" containerID="238a06fcd487895a6dc880506b7a15412bf57da87beaadca253fc1ce57c536c4"
Oct 30 19:51:31.806842 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:51:31.806824    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238a06fcd487895a6dc880506b7a15412bf57da87beaadca253fc1ce57c536c4\": container with ID starting with 238a06fcd487895a6dc880506b7a15412bf57da87beaadca253fc1ce57c536c4 not found: ID does not exist" containerID="238a06fcd487895a6dc880506b7a15412bf57da87beaadca253fc1ce57c536c4"
Oct 30 19:51:31.806906 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.806847    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238a06fcd487895a6dc880506b7a15412bf57da87beaadca253fc1ce57c536c4"} err="failed to get container status \"238a06fcd487895a6dc880506b7a15412bf57da87beaadca253fc1ce57c536c4\": rpc error: code = NotFound desc = could not find container \"238a06fcd487895a6dc880506b7a15412bf57da87beaadca253fc1ce57c536c4\": container with ID starting with 238a06fcd487895a6dc880506b7a15412bf57da87beaadca253fc1ce57c536c4 not found: ID does not exist"
Oct 30 19:51:31.837455 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.837381    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"]
Oct 30 19:51:31.840108 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.840071    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-040623-86f6f98d86-5pp5q"]
Oct 30 19:51:31.866486 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.866460    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"]
Oct 30 19:51:31.884563 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.884539    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-040623-6db97689bc-rh2kf"]
Oct 30 19:51:31.919481 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.919451    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"]
Oct 30 19:51:31.922325 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:31.922291    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-040623-d5446d769-57kwk"]
Oct 30 19:51:33.201982 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:33.201953    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3919c09d-9f26-4233-a9b9-6f38d6e39970" path="/var/lib/kubelet/pods/3919c09d-9f26-4233-a9b9-6f38d6e39970/volumes"
Oct 30 19:51:33.202413 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:33.202399    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a2ed94-fdfe-41ac-aced-bc1f2c326d06" path="/var/lib/kubelet/pods/54a2ed94-fdfe-41ac-aced-bc1f2c326d06/volumes"
Oct 30 19:51:33.202820 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:33.202806    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1feae8e-f123-45b9-a52b-86457fabe1ec" path="/var/lib/kubelet/pods/c1feae8e-f123-45b9-a52b-86457fabe1ec/volumes"
Oct 30 19:51:34.288839 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:51:34.288802    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:51:35.182066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:35.180921    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"]
Oct 30 19:51:40.182222 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.182175    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4" podUID="857bb812-0f79-433c-8f90-8a2a57232114" containerName="payload-processing" containerID="cri-o://9bde9fd0299ccac42a7e01887694117089735053f4dd5a17aa12326937ae0ac3" gracePeriod=25
Oct 30 19:51:40.523556 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.523532    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"
Oct 30 19:51:40.535336 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.535311    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/857bb812-0f79-433c-8f90-8a2a57232114-kube-api-access-dxdms\" (UniqueName: \"kubernetes.io/projected/857bb812-0f79-433c-8f90-8a2a57232114-kube-api-access-dxdms\") pod \"857bb812-0f79-433c-8f90-8a2a57232114\" (UID: \"857bb812-0f79-433c-8f90-8a2a57232114\") "
Oct 30 19:51:40.535563 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.535456    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/857bb812-0f79-433c-8f90-8a2a57232114-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/857bb812-0f79-433c-8f90-8a2a57232114-plugins-config-volume\") pod \"857bb812-0f79-433c-8f90-8a2a57232114\" (UID: \"857bb812-0f79-433c-8f90-8a2a57232114\") "
Oct 30 19:51:40.535793 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.535773    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/857bb812-0f79-433c-8f90-8a2a57232114-plugins-config-volume" pod "857bb812-0f79-433c-8f90-8a2a57232114" (UID: "857bb812-0f79-433c-8f90-8a2a57232114"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:40.535981 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.535966    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/857bb812-0f79-433c-8f90-8a2a57232114-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:40.537547 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.537518    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857bb812-0f79-433c-8f90-8a2a57232114-kube-api-access-dxdms" pod "857bb812-0f79-433c-8f90-8a2a57232114" (UID: "857bb812-0f79-433c-8f90-8a2a57232114"). InnerVolumeSpecName "kube-api-access-dxdms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:40.637488 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.637447    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxdms\" (UniqueName: \"kubernetes.io/projected/857bb812-0f79-433c-8f90-8a2a57232114-kube-api-access-dxdms\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:40.785785 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.785666    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"]
Oct 30 19:51:40.793215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793175    6030 cpu_manager.go:414] "containerMap: removing container" podUID="45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" containerName="istio-proxy"
Oct 30 19:51:40.793215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793212    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" containerName="istio-proxy"
Oct 30 19:51:40.793438 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793248    6030 cpu_manager.go:414] "containerMap: removing container" podUID="54a2ed94-fdfe-41ac-aced-bc1f2c326d06" containerName="payload-pre-processing"
Oct 30 19:51:40.793438 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793257    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a2ed94-fdfe-41ac-aced-bc1f2c326d06" containerName="payload-pre-processing"
Oct 30 19:51:40.793438 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793304    6030 cpu_manager.go:414] "containerMap: removing container" podUID="39e3cf00-165f-4cd0-b98e-4713a8ffbda1" containerName="maas-api"
Oct 30 19:51:40.793438 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793311    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e3cf00-165f-4cd0-b98e-4713a8ffbda1" containerName="maas-api"
Oct 30 19:51:40.793438 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793330    6030 cpu_manager.go:414] "containerMap: removing container" podUID="857bb812-0f79-433c-8f90-8a2a57232114" containerName="payload-processing"
Oct 30 19:51:40.793438 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793348    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="857bb812-0f79-433c-8f90-8a2a57232114" containerName="payload-processing"
Oct 30 19:51:40.793438 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793373    6030 cpu_manager.go:414] "containerMap: removing container" podUID="c1feae8e-f123-45b9-a52b-86457fabe1ec" containerName="maas-api"
Oct 30 19:51:40.793438 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793379    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1feae8e-f123-45b9-a52b-86457fabe1ec" containerName="maas-api"
Oct 30 19:51:40.793438 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793421    6030 cpu_manager.go:414] "containerMap: removing container" podUID="3919c09d-9f26-4233-a9b9-6f38d6e39970" containerName="payload-processing"
Oct 30 19:51:40.793438 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793436    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3919c09d-9f26-4233-a9b9-6f38d6e39970" containerName="payload-processing"
Oct 30 19:51:40.793911 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793734    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e3cf00-165f-4cd0-b98e-4713a8ffbda1" containerName="maas-api"
Oct 30 19:51:40.793911 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793749    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" containerName="istio-proxy"
Oct 30 19:51:40.793911 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793771    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" containerName="istio-proxy"
Oct 30 19:51:40.793911 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793789    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="857bb812-0f79-433c-8f90-8a2a57232114" containerName="payload-processing"
Oct 30 19:51:40.793911 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793813    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a2ed94-fdfe-41ac-aced-bc1f2c326d06" containerName="payload-pre-processing"
Oct 30 19:51:40.793911 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793832    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1feae8e-f123-45b9-a52b-86457fabe1ec" containerName="maas-api"
Oct 30 19:51:40.793911 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.793848    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="3919c09d-9f26-4233-a9b9-6f38d6e39970" containerName="payload-processing"
Oct 30 19:51:40.799154 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.798727    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.811892 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.811862    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"]
Oct 30 19:51:40.840867 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.840829    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-podinfo\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.841013 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.840995    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvht4\" (UniqueName: \"kubernetes.io/projected/842afcb7-9473-4e5a-b9fa-3486a236c1a5-kube-api-access-hvht4\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.841117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.841035    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istiod-ca-cert\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.841168 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.841150    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-data\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.841251 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.841222    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-workload-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.841294 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.841264    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-workload-certs\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.841294 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.841287    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-credential-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.841391 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.841372    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-envoy\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.841480 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.841465    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-token\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.882424 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.882388    6030 generic.go:356] "Generic (PLEG): container finished" podID="857bb812-0f79-433c-8f90-8a2a57232114" containerID="9bde9fd0299ccac42a7e01887694117089735053f4dd5a17aa12326937ae0ac3" exitCode=0
Oct 30 19:51:40.882928 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.882456    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"
Oct 30 19:51:40.882928 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.882509    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4" event={"ID":"857bb812-0f79-433c-8f90-8a2a57232114","Type":"ContainerDied","Data":"9bde9fd0299ccac42a7e01887694117089735053f4dd5a17aa12326937ae0ac3"}
Oct 30 19:51:40.882928 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.882563    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4" event={"ID":"857bb812-0f79-433c-8f90-8a2a57232114","Type":"ContainerDied","Data":"7d734aeab6410a6a8e01dfcf4cd39935996ee9e4b4abddbf6ccac3dcb7cd4f62"}
Oct 30 19:51:40.882928 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.882642    6030 scope.go:122] "RemoveContainer" containerID="9bde9fd0299ccac42a7e01887694117089735053f4dd5a17aa12326937ae0ac3"
Oct 30 19:51:40.885003 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.884982    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ec5e0f-openshift-default-dockercfg-2xxlz\""
Oct 30 19:51:40.898124 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.898098    6030 scope.go:122] "RemoveContainer" containerID="9bde9fd0299ccac42a7e01887694117089735053f4dd5a17aa12326937ae0ac3"
Oct 30 19:51:40.898455 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:51:40.898431    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bde9fd0299ccac42a7e01887694117089735053f4dd5a17aa12326937ae0ac3\": container with ID starting with 9bde9fd0299ccac42a7e01887694117089735053f4dd5a17aa12326937ae0ac3 not found: ID does not exist" containerID="9bde9fd0299ccac42a7e01887694117089735053f4dd5a17aa12326937ae0ac3"
Oct 30 19:51:40.898560 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.898463    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bde9fd0299ccac42a7e01887694117089735053f4dd5a17aa12326937ae0ac3"} err="failed to get container status \"9bde9fd0299ccac42a7e01887694117089735053f4dd5a17aa12326937ae0ac3\": rpc error: code = NotFound desc = could not find container \"9bde9fd0299ccac42a7e01887694117089735053f4dd5a17aa12326937ae0ac3\": container with ID starting with 9bde9fd0299ccac42a7e01887694117089735053f4dd5a17aa12326937ae0ac3 not found: ID does not exist"
Oct 30 19:51:40.917636 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.917608    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"]
Oct 30 19:51:40.927701 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.927675    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-78c8f75f4b-p4gs4"]
Oct 30 19:51:40.942978 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.942941    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-podinfo\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.943111 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.943082    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvht4\" (UniqueName: \"kubernetes.io/projected/842afcb7-9473-4e5a-b9fa-3486a236c1a5-kube-api-access-hvht4\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.943111 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.943109    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istiod-ca-cert\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.943213 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.943138    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-data\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.943213 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.943185    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-workload-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.943213 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.943210    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-workload-certs\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.943371 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.943241    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-credential-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.943371 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.943273    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-envoy\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.943371 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.943361    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-token\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.943693 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.943665    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-data\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.943853 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.943832    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-workload-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.943928 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.943861    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-credential-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.943928 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.943886    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istiod-ca-cert\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.943928 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.943899    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-workload-certs\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.945569 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.945547    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-podinfo\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.945703 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.945688    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-envoy\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.958958 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.958934    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-token\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:40.965564 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:40.965535    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvht4\" (UniqueName: \"kubernetes.io/projected/842afcb7-9473-4e5a-b9fa-3486a236c1a5-kube-api-access-hvht4\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:41.116631 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:41.116549    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:41.211153 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:41.211016    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="857bb812-0f79-433c-8f90-8a2a57232114" path="/var/lib/kubelet/pods/857bb812-0f79-433c-8f90-8a2a57232114/volumes"
Oct 30 19:51:41.406683 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:41.406654    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"]
Oct 30 19:51:41.407121 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:51:41.407093    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842afcb7_9473_4e5a_b9fa_3486a236c1a5.slice/crio-1b28129fa14aedacfc205c0d2078950d41198d897d8caa5de476005bca4ab117 WatchSource:0}: Error finding container 1b28129fa14aedacfc205c0d2078950d41198d897d8caa5de476005bca4ab117: Status 404 returned error can't find the container with id 1b28129fa14aedacfc205c0d2078950d41198d897d8caa5de476005bca4ab117
Oct 30 19:51:41.409808 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:41.409774    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:51:41.409870 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:41.409844    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:51:41.409911 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:41.409872    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:51:41.893446 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:41.893405    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6" event={"ID":"842afcb7-9473-4e5a-b9fa-3486a236c1a5","Type":"ContainerStarted","Data":"06c0f26176e3bcdb99c82bea78d7e361b7620f4c663923eb2beb3ba344b06ad0"}
Oct 30 19:51:41.893660 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:41.893474    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6" event={"ID":"842afcb7-9473-4e5a-b9fa-3486a236c1a5","Type":"ContainerStarted","Data":"1b28129fa14aedacfc205c0d2078950d41198d897d8caa5de476005bca4ab117"}
Oct 30 19:51:41.923312 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:41.923248    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6" podStartSLOduration=1.923228914 podStartE2EDuration="1.923228914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:41.920904975 +0000 UTC m=+1180.836511856" watchObservedRunningTime="2026-10-30 19:51:41.923228914 +0000 UTC m=+1180.838835772"
Oct 30 19:51:42.117123 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:42.117079    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:42.118797 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:42.118769    6030 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:51:42.118935 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:42.118820    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6" podUID="842afcb7-9473-4e5a-b9fa-3486a236c1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 19:51:43.117033 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:43.116996    6030 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:51:43.117523 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:43.117100    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6" podUID="842afcb7-9473-4e5a-b9fa-3486a236c1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 19:51:44.117275 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:44.117244    6030 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:51:44.117663 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:44.117296    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6" podUID="842afcb7-9473-4e5a-b9fa-3486a236c1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 19:51:45.158069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:45.157334    6030 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:51:45.158069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:45.157410    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6" podUID="842afcb7-9473-4e5a-b9fa-3486a236c1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 19:51:45.853091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:45.852429    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"]
Oct 30 19:51:45.853091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:45.852673    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq" podUID="e4869c19-e834-4312-a859-ca5af2c9cf63" containerName="istio-proxy" containerID="cri-o://c24a253ab8a3d3d8b95d310a6b64dd46c4bc88777c01fef9479254f82804da51" gracePeriod=30
Oct 30 19:51:46.131120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:46.129498    6030 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:51:46.131120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:46.129559    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6" podUID="842afcb7-9473-4e5a-b9fa-3486a236c1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 19:51:47.117119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:47.117084    6030 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:51:47.117502 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:47.117154    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6" podUID="842afcb7-9473-4e5a-b9fa-3486a236c1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 19:51:48.122341 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:48.122311    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:48.122812 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:48.122705    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:48.123390 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:48.123371    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:51:51.092812 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.092775    6030 generic.go:356] "Generic (PLEG): container finished" podID="e4869c19-e834-4312-a859-ca5af2c9cf63" containerID="c24a253ab8a3d3d8b95d310a6b64dd46c4bc88777c01fef9479254f82804da51" exitCode=0
Oct 30 19:51:51.093310 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.092854    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq" event={"ID":"e4869c19-e834-4312-a859-ca5af2c9cf63","Type":"ContainerDied","Data":"c24a253ab8a3d3d8b95d310a6b64dd46c4bc88777c01fef9479254f82804da51"}
Oct 30 19:51:51.138659 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.138602    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:51:51.228424 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.228396    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4869c19-e834-4312-a859-ca5af2c9cf63-kube-api-access-hwks6\" (UniqueName: \"kubernetes.io/projected/e4869c19-e834-4312-a859-ca5af2c9cf63-kube-api-access-hwks6\") pod \"e4869c19-e834-4312-a859-ca5af2c9cf63\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") "
Oct 30 19:51:51.228597 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.228559    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-workload-socket\") pod \"e4869c19-e834-4312-a859-ca5af2c9cf63\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") "
Oct 30 19:51:51.228597 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.228591    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-workload-certs\") pod \"e4869c19-e834-4312-a859-ca5af2c9cf63\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") "
Oct 30 19:51:51.228716 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.228644    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-credential-socket\") pod \"e4869c19-e834-4312-a859-ca5af2c9cf63\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") "
Oct 30 19:51:51.228716 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.228681    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-data\") pod \"e4869c19-e834-4312-a859-ca5af2c9cf63\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") "
Oct 30 19:51:51.228824 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.228744    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-envoy\") pod \"e4869c19-e834-4312-a859-ca5af2c9cf63\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") "
Oct 30 19:51:51.228878 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.228827    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-token\" (UniqueName: \"kubernetes.io/projected/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-token\") pod \"e4869c19-e834-4312-a859-ca5af2c9cf63\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") "
Oct 30 19:51:51.228878 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.228855    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4869c19-e834-4312-a859-ca5af2c9cf63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4869c19-e834-4312-a859-ca5af2c9cf63-istiod-ca-cert\") pod \"e4869c19-e834-4312-a859-ca5af2c9cf63\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") "
Oct 30 19:51:51.228983 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.228904    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-workload-socket" pod "e4869c19-e834-4312-a859-ca5af2c9cf63" (UID: "e4869c19-e834-4312-a859-ca5af2c9cf63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:51.229080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.228982    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-credential-socket" pod "e4869c19-e834-4312-a859-ca5af2c9cf63" (UID: "e4869c19-e834-4312-a859-ca5af2c9cf63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:51.229580 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.229243    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-workload-certs" pod "e4869c19-e834-4312-a859-ca5af2c9cf63" (UID: "e4869c19-e834-4312-a859-ca5af2c9cf63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:51.229872 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.229851    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-data" pod "e4869c19-e834-4312-a859-ca5af2c9cf63" (UID: "e4869c19-e834-4312-a859-ca5af2c9cf63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:51.230604 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.230578    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-podinfo\") pod \"e4869c19-e834-4312-a859-ca5af2c9cf63\" (UID: \"e4869c19-e834-4312-a859-ca5af2c9cf63\") "
Oct 30 19:51:51.231410 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.231384    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:51.231511 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.231413    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:51.231511 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.231425    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:51.231511 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.231437    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:51.233632 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.232922    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4869c19-e834-4312-a859-ca5af2c9cf63-istiod-ca-cert" pod "e4869c19-e834-4312-a859-ca5af2c9cf63" (UID: "e4869c19-e834-4312-a859-ca5af2c9cf63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:51.234201 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.234165    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-podinfo" pod "e4869c19-e834-4312-a859-ca5af2c9cf63" (UID: "e4869c19-e834-4312-a859-ca5af2c9cf63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:51:51.234201 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.234165    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-token" pod "e4869c19-e834-4312-a859-ca5af2c9cf63" (UID: "e4869c19-e834-4312-a859-ca5af2c9cf63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:51.236337 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.236311    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-envoy" pod "e4869c19-e834-4312-a859-ca5af2c9cf63" (UID: "e4869c19-e834-4312-a859-ca5af2c9cf63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:51.243572 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.243532    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4869c19-e834-4312-a859-ca5af2c9cf63-kube-api-access-hwks6" pod "e4869c19-e834-4312-a859-ca5af2c9cf63" (UID: "e4869c19-e834-4312-a859-ca5af2c9cf63"). InnerVolumeSpecName "kube-api-access-hwks6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:51.332669 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.332633    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwks6\" (UniqueName: \"kubernetes.io/projected/e4869c19-e834-4312-a859-ca5af2c9cf63-kube-api-access-hwks6\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:51.332669 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.332664    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:51.332669 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.332677    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:51.332972 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.332689    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4869c19-e834-4312-a859-ca5af2c9cf63-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:51.332972 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:51.332703    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4869c19-e834-4312-a859-ca5af2c9cf63-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:52.100799 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:52.100763    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"
Oct 30 19:51:52.101240 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:52.100762    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq" event={"ID":"e4869c19-e834-4312-a859-ca5af2c9cf63","Type":"ContainerDied","Data":"07895e1344529043a3e47276ee04d9a5977bd7293011c6260867dd6955737969"}
Oct 30 19:51:52.101240 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:52.100912    6030 scope.go:122] "RemoveContainer" containerID="c24a253ab8a3d3d8b95d310a6b64dd46c4bc88777c01fef9479254f82804da51"
Oct 30 19:51:52.128406 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:52.128380    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"]
Oct 30 19:51:52.135598 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:52.135574    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-799mq"]
Oct 30 19:51:53.209261 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:53.209226    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4869c19-e834-4312-a859-ca5af2c9cf63" path="/var/lib/kubelet/pods/e4869c19-e834-4312-a859-ca5af2c9cf63/volumes"
Oct 30 19:51:58.028430 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:58.028365    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"]
Oct 30 19:51:58.033614 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:58.033576    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4daf28a3-r2nqz"]
Oct 30 19:51:59.202939 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:51:59.202903    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93727270-f8e9-4880-8cac-7aa7a5f89f9f" path="/var/lib/kubelet/pods/93727270-f8e9-4880-8cac-7aa7a5f89f9f/volumes"
Oct 30 19:52:00.673218 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:00.673194    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s_2a0204dc-bca5-497c-8d32-1057e9e0af46/payload-processing/0.log"
Oct 30 19:52:01.392680 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.392646    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc_8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5/payload-processing/0.log"
Oct 30 19:52:01.582814 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.582779    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"]
Oct 30 19:52:01.583391 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.583369    6030 cpu_manager.go:414] "containerMap: removing container" podUID="e4869c19-e834-4312-a859-ca5af2c9cf63" containerName="istio-proxy"
Oct 30 19:52:01.583391 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.583389    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4869c19-e834-4312-a859-ca5af2c9cf63" containerName="istio-proxy"
Oct 30 19:52:01.583595 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.583400    6030 cpu_manager.go:414] "containerMap: removing container" podUID="45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" containerName="istio-proxy"
Oct 30 19:52:01.583595 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.583405    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45eef3eb-e56b-4e28-ae5c-02b5eeae51b8" containerName="istio-proxy"
Oct 30 19:52:01.583595 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.583515    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4869c19-e834-4312-a859-ca5af2c9cf63" containerName="istio-proxy"
Oct 30 19:52:01.589579 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.589549    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:52:01.590375 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.590354    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc_8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5/payload-processing/0.log"
Oct 30 19:52:01.609095 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.604100    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"]
Oct 30 19:52:01.642000 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.640317    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-b17e94-serving-cert\""
Oct 30 19:52:01.649756 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.649683    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2170ff4d-355b-4c88-ad37-cfd014e9bb17-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb\" (UID: \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:52:01.649756 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.649735    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2170ff4d-355b-4c88-ad37-cfd014e9bb17-metrics-certs\") pod \"maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb\" (UID: \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:52:01.649992 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.649967    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljrj5\" (UniqueName: \"kubernetes.io/projected/2170ff4d-355b-4c88-ad37-cfd014e9bb17-kube-api-access-ljrj5\") pod \"maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb\" (UID: \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:52:01.650030 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.650021    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2170ff4d-355b-4c88-ad37-cfd014e9bb17-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb\" (UID: \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:52:01.755078 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.753558    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljrj5\" (UniqueName: \"kubernetes.io/projected/2170ff4d-355b-4c88-ad37-cfd014e9bb17-kube-api-access-ljrj5\") pod \"maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb\" (UID: \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:52:01.755078 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.753635    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2170ff4d-355b-4c88-ad37-cfd014e9bb17-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb\" (UID: \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:52:01.755078 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.753753    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2170ff4d-355b-4c88-ad37-cfd014e9bb17-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb\" (UID: \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:52:01.755078 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.753797    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2170ff4d-355b-4c88-ad37-cfd014e9bb17-metrics-certs\") pod \"maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb\" (UID: \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:52:01.775113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.755880    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2170ff4d-355b-4c88-ad37-cfd014e9bb17-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb\" (UID: \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:52:01.775113 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:52:01.756282    6030 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-serving-cert: secret "maas-api-e2e-praxis-a-b17e94-serving-cert" not found
Oct 30 19:52:01.775113 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:52:01.756346    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2170ff4d-355b-4c88-ad37-cfd014e9bb17-maas-api-tls podName:2170ff4d-355b-4c88-ad37-cfd014e9bb17 nodeName:}" failed. No retries permitted until 2026-10-30 19:52:02.256325416 +0000 UTC m=+1201.171932264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2170ff4d-355b-4c88-ad37-cfd014e9bb17-maas-api-tls") pod "maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb" (UID: "2170ff4d-355b-4c88-ad37-cfd014e9bb17") : secret "maas-api-e2e-praxis-a-b17e94-serving-cert" not found
Oct 30 19:52:01.785739 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.785702    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2170ff4d-355b-4c88-ad37-cfd014e9bb17-metrics-certs\") pod \"maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb\" (UID: \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:52:01.809813 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.809778    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljrj5\" (UniqueName: \"kubernetes.io/projected/2170ff4d-355b-4c88-ad37-cfd014e9bb17-kube-api-access-ljrj5\") pod \"maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb\" (UID: \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:52:01.930129 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.930031    6030 scope.go:122] "RemoveContainer" containerID="c8f141f95ecb88d14620e5b54b3c21ccff310961ab2c54436e1e9074b72cb957"
Oct 30 19:52:01.942630 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.942600    6030 scope.go:122] "RemoveContainer" containerID="c1f56dd77a9c0c832ef4d035ad4aae87a81131544d332c46d5401c983b6283b4"
Oct 30 19:52:01.955081 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:01.955015    6030 scope.go:122] "RemoveContainer" containerID="e898e0c10951c4ec750e444bef2427a87d3f2f7ea06ed80d1ddd05caa0f9859e"
Oct 30 19:52:02.259876 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:02.259794    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2170ff4d-355b-4c88-ad37-cfd014e9bb17-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb\" (UID: \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:52:02.265561 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:02.265533    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2170ff4d-355b-4c88-ad37-cfd014e9bb17-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb\" (UID: \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:52:02.520126 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:02.514298    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:52:02.934028 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:02.933988    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"]
Oct 30 19:52:02.936267 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:52:02.936231    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2170ff4d_355b_4c88_ad37_cfd014e9bb17.slice/crio-b6a2fdd8427acd9e63127bd29268b1bfcd7eb0381ede74d2ffbf312964281b5f WatchSource:0}: Error finding container b6a2fdd8427acd9e63127bd29268b1bfcd7eb0381ede74d2ffbf312964281b5f: Status 404 returned error can't find the container with id b6a2fdd8427acd9e63127bd29268b1bfcd7eb0381ede74d2ffbf312964281b5f
Oct 30 19:52:02.938311 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:02.938292    6030 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:52:03.250036 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:03.249950    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb" event={"ID":"2170ff4d-355b-4c88-ad37-cfd014e9bb17","Type":"ContainerStarted","Data":"b6a2fdd8427acd9e63127bd29268b1bfcd7eb0381ede74d2ffbf312964281b5f"}
Oct 30 19:52:04.256800 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:04.256761    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb" event={"ID":"2170ff4d-355b-4c88-ad37-cfd014e9bb17","Type":"ContainerStarted","Data":"5b98b7a86280e0793baede3cf76ed20102042ef2942e85fbcf0e3f6d6a37590a"}
Oct 30 19:52:04.257224 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:04.256849    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:52:04.280426 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:04.280361    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb" podStartSLOduration=2.8424346700000003 podStartE2EDuration="3.280341731s" totalImagesPullingTime="437.907061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:04.279037518 +0000 UTC m=+1203.194644372" watchObservedRunningTime="2026-10-30 19:52:04.280341731 +0000 UTC m=+1203.195948588"
Oct 30 19:52:04.291636 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:52:04.291604    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:52:10.270546 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:10.270517    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:52:34.292127 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:52:34.292085    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:52:35.745862 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:35.745829    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"]
Oct 30 19:52:35.746283 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:35.746136    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt" podUID="46f60b19-53a4-4b56-8a23-20501f978935" containerName="maas-api" containerID="cri-o://60e7dac58ba33e547f055a804101a162b010c65e55e349804c5a1404900e3665" gracePeriod=30
Oct 30 19:52:35.973019 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:35.972986    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"]
Oct 30 19:52:35.973493 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:35.973444    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc" podUID="1f131160-44fc-416c-9f1a-36786452d4db" containerName="payload-processing" containerID="cri-o://6ea841a96da1b6f665f981f9b44de70f17ea15e35e5be6e2742c204c3d285d29" gracePeriod=30
Oct 30 19:52:36.128506 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.128270    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:52:36.254607 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.254576    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46f60b19-53a4-4b56-8a23-20501f978935-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f60b19-53a4-4b56-8a23-20501f978935-openshift-service-ca\") pod \"46f60b19-53a4-4b56-8a23-20501f978935\" (UID: \"46f60b19-53a4-4b56-8a23-20501f978935\") "
Oct 30 19:52:36.254806 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.254620    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46f60b19-53a4-4b56-8a23-20501f978935-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46f60b19-53a4-4b56-8a23-20501f978935-metrics-certs\") pod \"46f60b19-53a4-4b56-8a23-20501f978935\" (UID: \"46f60b19-53a4-4b56-8a23-20501f978935\") "
Oct 30 19:52:36.254806 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.254708    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46f60b19-53a4-4b56-8a23-20501f978935-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f60b19-53a4-4b56-8a23-20501f978935-maas-api-tls\") pod \"46f60b19-53a4-4b56-8a23-20501f978935\" (UID: \"46f60b19-53a4-4b56-8a23-20501f978935\") "
Oct 30 19:52:36.254940 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.254830    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46f60b19-53a4-4b56-8a23-20501f978935-kube-api-access-pzfnc\" (UniqueName: \"kubernetes.io/projected/46f60b19-53a4-4b56-8a23-20501f978935-kube-api-access-pzfnc\") pod \"46f60b19-53a4-4b56-8a23-20501f978935\" (UID: \"46f60b19-53a4-4b56-8a23-20501f978935\") "
Oct 30 19:52:36.255509 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.255472    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f60b19-53a4-4b56-8a23-20501f978935-openshift-service-ca" pod "46f60b19-53a4-4b56-8a23-20501f978935" (UID: "46f60b19-53a4-4b56-8a23-20501f978935"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:36.255791 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.255767    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f60b19-53a4-4b56-8a23-20501f978935-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:36.258165 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.258136    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46f60b19-53a4-4b56-8a23-20501f978935-maas-api-tls" pod "46f60b19-53a4-4b56-8a23-20501f978935" (UID: "46f60b19-53a4-4b56-8a23-20501f978935"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:36.259263 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.259238    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f60b19-53a4-4b56-8a23-20501f978935-kube-api-access-pzfnc" pod "46f60b19-53a4-4b56-8a23-20501f978935" (UID: "46f60b19-53a4-4b56-8a23-20501f978935"). InnerVolumeSpecName "kube-api-access-pzfnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:36.266265 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.266202    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46f60b19-53a4-4b56-8a23-20501f978935-metrics-certs" pod "46f60b19-53a4-4b56-8a23-20501f978935" (UID: "46f60b19-53a4-4b56-8a23-20501f978935"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:36.365093 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.360157    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46f60b19-53a4-4b56-8a23-20501f978935-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:36.365093 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.360194    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f60b19-53a4-4b56-8a23-20501f978935-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:36.365093 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.360212    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzfnc\" (UniqueName: \"kubernetes.io/projected/46f60b19-53a4-4b56-8a23-20501f978935-kube-api-access-pzfnc\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:36.471071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.470556    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"
Oct 30 19:52:36.564297 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.562286    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f131160-44fc-416c-9f1a-36786452d4db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f131160-44fc-416c-9f1a-36786452d4db-plugins-config-volume\") pod \"1f131160-44fc-416c-9f1a-36786452d4db\" (UID: \"1f131160-44fc-416c-9f1a-36786452d4db\") "
Oct 30 19:52:36.564297 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.562455    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f131160-44fc-416c-9f1a-36786452d4db-kube-api-access-zlt7z\" (UniqueName: \"kubernetes.io/projected/1f131160-44fc-416c-9f1a-36786452d4db-kube-api-access-zlt7z\") pod \"1f131160-44fc-416c-9f1a-36786452d4db\" (UID: \"1f131160-44fc-416c-9f1a-36786452d4db\") "
Oct 30 19:52:36.564877 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.564593    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f131160-44fc-416c-9f1a-36786452d4db-plugins-config-volume" pod "1f131160-44fc-416c-9f1a-36786452d4db" (UID: "1f131160-44fc-416c-9f1a-36786452d4db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:36.566021 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.565953    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f131160-44fc-416c-9f1a-36786452d4db-kube-api-access-zlt7z" pod "1f131160-44fc-416c-9f1a-36786452d4db" (UID: "1f131160-44fc-416c-9f1a-36786452d4db"). InnerVolumeSpecName "kube-api-access-zlt7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:36.606304 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.606265    6030 generic.go:356] "Generic (PLEG): container finished" podID="46f60b19-53a4-4b56-8a23-20501f978935" containerID="60e7dac58ba33e547f055a804101a162b010c65e55e349804c5a1404900e3665" exitCode=0
Oct 30 19:52:36.606496 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.606327    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"
Oct 30 19:52:36.606496 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.606349    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt" event={"ID":"46f60b19-53a4-4b56-8a23-20501f978935","Type":"ContainerDied","Data":"60e7dac58ba33e547f055a804101a162b010c65e55e349804c5a1404900e3665"}
Oct 30 19:52:36.606496 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.606418    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt" event={"ID":"46f60b19-53a4-4b56-8a23-20501f978935","Type":"ContainerDied","Data":"b0c8d77a24c05b24090949717050081b0f0757617c9e7949fff5a16190774612"}
Oct 30 19:52:36.606496 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.606442    6030 scope.go:122] "RemoveContainer" containerID="60e7dac58ba33e547f055a804101a162b010c65e55e349804c5a1404900e3665"
Oct 30 19:52:36.616126 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.612316    6030 generic.go:356] "Generic (PLEG): container finished" podID="1f131160-44fc-416c-9f1a-36786452d4db" containerID="6ea841a96da1b6f665f981f9b44de70f17ea15e35e5be6e2742c204c3d285d29" exitCode=0
Oct 30 19:52:36.616126 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.612376    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"
Oct 30 19:52:36.616126 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.612402    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc" event={"ID":"1f131160-44fc-416c-9f1a-36786452d4db","Type":"ContainerDied","Data":"6ea841a96da1b6f665f981f9b44de70f17ea15e35e5be6e2742c204c3d285d29"}
Oct 30 19:52:36.616126 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.612438    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc" event={"ID":"1f131160-44fc-416c-9f1a-36786452d4db","Type":"ContainerDied","Data":"bb34b8f74d38555d68152649d627a447464137b42421d46a6cc02c521c7583ad"}
Oct 30 19:52:36.624006 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.623975    6030 scope.go:122] "RemoveContainer" containerID="60e7dac58ba33e547f055a804101a162b010c65e55e349804c5a1404900e3665"
Oct 30 19:52:36.624360 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:52:36.624337    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e7dac58ba33e547f055a804101a162b010c65e55e349804c5a1404900e3665\": container with ID starting with 60e7dac58ba33e547f055a804101a162b010c65e55e349804c5a1404900e3665 not found: ID does not exist" containerID="60e7dac58ba33e547f055a804101a162b010c65e55e349804c5a1404900e3665"
Oct 30 19:52:36.624452 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.624369    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e7dac58ba33e547f055a804101a162b010c65e55e349804c5a1404900e3665"} err="failed to get container status \"60e7dac58ba33e547f055a804101a162b010c65e55e349804c5a1404900e3665\": rpc error: code = NotFound desc = could not find container \"60e7dac58ba33e547f055a804101a162b010c65e55e349804c5a1404900e3665\": container with ID starting with 60e7dac58ba33e547f055a804101a162b010c65e55e349804c5a1404900e3665 not found: ID does not exist"
Oct 30 19:52:36.624452 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.624385    6030 scope.go:122] "RemoveContainer" containerID="6ea841a96da1b6f665f981f9b44de70f17ea15e35e5be6e2742c204c3d285d29"
Oct 30 19:52:36.640111 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.636768    6030 scope.go:122] "RemoveContainer" containerID="6ea841a96da1b6f665f981f9b44de70f17ea15e35e5be6e2742c204c3d285d29"
Oct 30 19:52:36.640111 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:52:36.637122    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea841a96da1b6f665f981f9b44de70f17ea15e35e5be6e2742c204c3d285d29\": container with ID starting with 6ea841a96da1b6f665f981f9b44de70f17ea15e35e5be6e2742c204c3d285d29 not found: ID does not exist" containerID="6ea841a96da1b6f665f981f9b44de70f17ea15e35e5be6e2742c204c3d285d29"
Oct 30 19:52:36.640111 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.637155    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea841a96da1b6f665f981f9b44de70f17ea15e35e5be6e2742c204c3d285d29"} err="failed to get container status \"6ea841a96da1b6f665f981f9b44de70f17ea15e35e5be6e2742c204c3d285d29\": rpc error: code = NotFound desc = could not find container \"6ea841a96da1b6f665f981f9b44de70f17ea15e35e5be6e2742c204c3d285d29\": container with ID starting with 6ea841a96da1b6f665f981f9b44de70f17ea15e35e5be6e2742c204c3d285d29 not found: ID does not exist"
Oct 30 19:52:36.653415 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.653385    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"]
Oct 30 19:52:36.661747 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.661717    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f2325-65b4ff6d8c-rtdpt"]
Oct 30 19:52:36.664024 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.664005    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlt7z\" (UniqueName: \"kubernetes.io/projected/1f131160-44fc-416c-9f1a-36786452d4db-kube-api-access-zlt7z\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:36.664134 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.664026    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f131160-44fc-416c-9f1a-36786452d4db-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:36.713543 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.713512    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"]
Oct 30 19:52:36.746437 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:36.746403    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f2325-74f4586c46-k8dqc"]
Oct 30 19:52:37.203166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:37.203138    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f131160-44fc-416c-9f1a-36786452d4db" path="/var/lib/kubelet/pods/1f131160-44fc-416c-9f1a-36786452d4db/volumes"
Oct 30 19:52:37.203592 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:37.203578    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46f60b19-53a4-4b56-8a23-20501f978935" path="/var/lib/kubelet/pods/46f60b19-53a4-4b56-8a23-20501f978935/volumes"
Oct 30 19:52:37.949528 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:37.949491    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"]
Oct 30 19:52:37.950323 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:37.950291    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s" podUID="2a0204dc-bca5-497c-8d32-1057e9e0af46" containerName="payload-processing" containerID="cri-o://99a0022360aebe6aa27184ce417944fb549dce392057563aba734eaed55a450c" gracePeriod=30
Oct 30 19:52:38.552561 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:38.552528    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"
Oct 30 19:52:38.632515 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:38.632481    6030 generic.go:356] "Generic (PLEG): container finished" podID="2a0204dc-bca5-497c-8d32-1057e9e0af46" containerID="99a0022360aebe6aa27184ce417944fb549dce392057563aba734eaed55a450c" exitCode=0
Oct 30 19:52:38.632733 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:38.632529    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"
Oct 30 19:52:38.632733 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:38.632571    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s" event={"ID":"2a0204dc-bca5-497c-8d32-1057e9e0af46","Type":"ContainerDied","Data":"99a0022360aebe6aa27184ce417944fb549dce392057563aba734eaed55a450c"}
Oct 30 19:52:38.632733 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:38.632636    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s" event={"ID":"2a0204dc-bca5-497c-8d32-1057e9e0af46","Type":"ContainerDied","Data":"e3ea015f95d8bbbf8ed38d7f4dae42745bdd93956e82a5b31aad0da8b3dbebf7"}
Oct 30 19:52:38.632733 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:38.632664    6030 scope.go:122] "RemoveContainer" containerID="99a0022360aebe6aa27184ce417944fb549dce392057563aba734eaed55a450c"
Oct 30 19:52:38.643525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:38.643504    6030 scope.go:122] "RemoveContainer" containerID="99a0022360aebe6aa27184ce417944fb549dce392057563aba734eaed55a450c"
Oct 30 19:52:38.643845 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:52:38.643820    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a0022360aebe6aa27184ce417944fb549dce392057563aba734eaed55a450c\": container with ID starting with 99a0022360aebe6aa27184ce417944fb549dce392057563aba734eaed55a450c not found: ID does not exist" containerID="99a0022360aebe6aa27184ce417944fb549dce392057563aba734eaed55a450c"
Oct 30 19:52:38.643910 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:38.643857    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a0022360aebe6aa27184ce417944fb549dce392057563aba734eaed55a450c"} err="failed to get container status \"99a0022360aebe6aa27184ce417944fb549dce392057563aba734eaed55a450c\": rpc error: code = NotFound desc = could not find container \"99a0022360aebe6aa27184ce417944fb549dce392057563aba734eaed55a450c\": container with ID starting with 99a0022360aebe6aa27184ce417944fb549dce392057563aba734eaed55a450c not found: ID does not exist"
Oct 30 19:52:38.753099 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:38.750901    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a0204dc-bca5-497c-8d32-1057e9e0af46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0204dc-bca5-497c-8d32-1057e9e0af46-plugins-config-volume\") pod \"2a0204dc-bca5-497c-8d32-1057e9e0af46\" (UID: \"2a0204dc-bca5-497c-8d32-1057e9e0af46\") "
Oct 30 19:52:38.753099 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:38.751017    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0204dc-bca5-497c-8d32-1057e9e0af46-kube-api-access-rsqwh\" (UniqueName: \"kubernetes.io/projected/2a0204dc-bca5-497c-8d32-1057e9e0af46-kube-api-access-rsqwh\") pod \"2a0204dc-bca5-497c-8d32-1057e9e0af46\" (UID: \"2a0204dc-bca5-497c-8d32-1057e9e0af46\") "
Oct 30 19:52:38.759577 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:38.759506    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0204dc-bca5-497c-8d32-1057e9e0af46-plugins-config-volume" pod "2a0204dc-bca5-497c-8d32-1057e9e0af46" (UID: "2a0204dc-bca5-497c-8d32-1057e9e0af46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:38.761579 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:38.760459    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0204dc-bca5-497c-8d32-1057e9e0af46-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:38.798068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:38.795510    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0204dc-bca5-497c-8d32-1057e9e0af46-kube-api-access-rsqwh" pod "2a0204dc-bca5-497c-8d32-1057e9e0af46" (UID: "2a0204dc-bca5-497c-8d32-1057e9e0af46"). InnerVolumeSpecName "kube-api-access-rsqwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:38.862111 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:38.862077    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsqwh\" (UniqueName: \"kubernetes.io/projected/2a0204dc-bca5-497c-8d32-1057e9e0af46-kube-api-access-rsqwh\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:38.977968 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:38.977932    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"]
Oct 30 19:52:38.982422 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:38.982392    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-913351-5fdf8855f8-cmg2s"]
Oct 30 19:52:39.201434 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.201402    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a0204dc-bca5-497c-8d32-1057e9e0af46" path="/var/lib/kubelet/pods/2a0204dc-bca5-497c-8d32-1057e9e0af46/volumes"
Oct 30 19:52:39.849727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.849698    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"]
Oct 30 19:52:39.850316 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.850297    6030 cpu_manager.go:414] "containerMap: removing container" podUID="46f60b19-53a4-4b56-8a23-20501f978935" containerName="maas-api"
Oct 30 19:52:39.850429 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.850318    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46f60b19-53a4-4b56-8a23-20501f978935" containerName="maas-api"
Oct 30 19:52:39.850429 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.850388    6030 cpu_manager.go:414] "containerMap: removing container" podUID="2a0204dc-bca5-497c-8d32-1057e9e0af46" containerName="payload-processing"
Oct 30 19:52:39.850429 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.850397    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0204dc-bca5-497c-8d32-1057e9e0af46" containerName="payload-processing"
Oct 30 19:52:39.850429 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.850411    6030 cpu_manager.go:414] "containerMap: removing container" podUID="1f131160-44fc-416c-9f1a-36786452d4db" containerName="payload-processing"
Oct 30 19:52:39.850429 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.850417    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f131160-44fc-416c-9f1a-36786452d4db" containerName="payload-processing"
Oct 30 19:52:39.850665 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.850551    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0204dc-bca5-497c-8d32-1057e9e0af46" containerName="payload-processing"
Oct 30 19:52:39.850665 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.850564    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f131160-44fc-416c-9f1a-36786452d4db" containerName="payload-processing"
Oct 30 19:52:39.850665 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.850579    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="46f60b19-53a4-4b56-8a23-20501f978935" containerName="maas-api"
Oct 30 19:52:39.855743 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.855719    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:52:39.868755 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.868709    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-f03120-serving-cert\""
Oct 30 19:52:39.870510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.870464    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"]
Oct 30 19:52:39.874942 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.874909    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zddnw\" (UniqueName: \"kubernetes.io/projected/f758324a-5fb7-4274-b85d-82767be18e82-kube-api-access-zddnw\") pod \"maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6\" (UID: \"f758324a-5fb7-4274-b85d-82767be18e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:52:39.875105 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.874971    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f758324a-5fb7-4274-b85d-82767be18e82-metrics-certs\") pod \"maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6\" (UID: \"f758324a-5fb7-4274-b85d-82767be18e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:52:39.875171 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.875138    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f758324a-5fb7-4274-b85d-82767be18e82-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6\" (UID: \"f758324a-5fb7-4274-b85d-82767be18e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:52:39.875258 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.875236    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f758324a-5fb7-4274-b85d-82767be18e82-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6\" (UID: \"f758324a-5fb7-4274-b85d-82767be18e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:52:39.977679 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.977646    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f758324a-5fb7-4274-b85d-82767be18e82-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6\" (UID: \"f758324a-5fb7-4274-b85d-82767be18e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:52:39.995153 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.977776    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f758324a-5fb7-4274-b85d-82767be18e82-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6\" (UID: \"f758324a-5fb7-4274-b85d-82767be18e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:52:39.995153 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.977905    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zddnw\" (UniqueName: \"kubernetes.io/projected/f758324a-5fb7-4274-b85d-82767be18e82-kube-api-access-zddnw\") pod \"maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6\" (UID: \"f758324a-5fb7-4274-b85d-82767be18e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:52:39.995153 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:39.977950    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f758324a-5fb7-4274-b85d-82767be18e82-metrics-certs\") pod \"maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6\" (UID: \"f758324a-5fb7-4274-b85d-82767be18e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:52:40.031071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:40.012687    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f758324a-5fb7-4274-b85d-82767be18e82-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6\" (UID: \"f758324a-5fb7-4274-b85d-82767be18e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:52:40.031071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:40.014564    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f758324a-5fb7-4274-b85d-82767be18e82-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6\" (UID: \"f758324a-5fb7-4274-b85d-82767be18e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:52:40.031071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:40.015469    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f758324a-5fb7-4274-b85d-82767be18e82-metrics-certs\") pod \"maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6\" (UID: \"f758324a-5fb7-4274-b85d-82767be18e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:52:40.040183 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:40.039479    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zddnw\" (UniqueName: \"kubernetes.io/projected/f758324a-5fb7-4274-b85d-82767be18e82-kube-api-access-zddnw\") pod \"maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6\" (UID: \"f758324a-5fb7-4274-b85d-82767be18e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:52:40.177658 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:40.177628    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:52:40.479175 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:40.479142    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"]
Oct 30 19:52:40.480221 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:52:40.480188    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf758324a_5fb7_4274_b85d_82767be18e82.slice/crio-bd82b1fa5f855e16f05a5fd1cc094f7e6f46bdd5b221bfce6b39b08bb055c7b4 WatchSource:0}: Error finding container bd82b1fa5f855e16f05a5fd1cc094f7e6f46bdd5b221bfce6b39b08bb055c7b4: Status 404 returned error can't find the container with id bd82b1fa5f855e16f05a5fd1cc094f7e6f46bdd5b221bfce6b39b08bb055c7b4
Oct 30 19:52:40.685069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:40.677159    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6" event={"ID":"f758324a-5fb7-4274-b85d-82767be18e82","Type":"ContainerStarted","Data":"bd82b1fa5f855e16f05a5fd1cc094f7e6f46bdd5b221bfce6b39b08bb055c7b4"}
Oct 30 19:52:41.684772 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:41.684737    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6" event={"ID":"f758324a-5fb7-4274-b85d-82767be18e82","Type":"ContainerStarted","Data":"a2dc1d26a67c30f3a2894d13f9aa3247c6a26b9c1edac18c5c0e61009a0a4660"}
Oct 30 19:52:41.685195 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:41.684854    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:52:41.716447 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:41.716398    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6" podStartSLOduration=2.217805798 podStartE2EDuration="2.716383104s" totalImagesPullingTime="498.577306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:41.713583781 +0000 UTC m=+1240.629190636" watchObservedRunningTime="2026-10-30 19:52:41.716383104 +0000 UTC m=+1240.631990003"
Oct 30 19:52:42.075841 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:42.075760    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"]
Oct 30 19:52:42.082036 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:42.082005    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"
Oct 30 19:52:42.094112 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:42.094086    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"]
Oct 30 19:52:42.094582 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:42.094559    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-426bab-dockercfg-2hkjt\""
Oct 30 19:52:42.094824 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:42.094810    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-426bab\""
Oct 30 19:52:42.209071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:42.205671    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efb9f1a-3f05-4ddd-9ab5-7086f717524c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4\" (UID: \"8efb9f1a-3f05-4ddd-9ab5-7086f717524c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"
Oct 30 19:52:42.209071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:42.205735    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6j4m\" (UniqueName: \"kubernetes.io/projected/8efb9f1a-3f05-4ddd-9ab5-7086f717524c-kube-api-access-b6j4m\") pod \"payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4\" (UID: \"8efb9f1a-3f05-4ddd-9ab5-7086f717524c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"
Oct 30 19:52:42.312100 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:42.307999    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efb9f1a-3f05-4ddd-9ab5-7086f717524c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4\" (UID: \"8efb9f1a-3f05-4ddd-9ab5-7086f717524c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"
Oct 30 19:52:42.312100 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:42.308099    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6j4m\" (UniqueName: \"kubernetes.io/projected/8efb9f1a-3f05-4ddd-9ab5-7086f717524c-kube-api-access-b6j4m\") pod \"payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4\" (UID: \"8efb9f1a-3f05-4ddd-9ab5-7086f717524c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"
Oct 30 19:52:42.312100 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:42.309779    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efb9f1a-3f05-4ddd-9ab5-7086f717524c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4\" (UID: \"8efb9f1a-3f05-4ddd-9ab5-7086f717524c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"
Oct 30 19:52:42.354089 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:42.353955    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6j4m\" (UniqueName: \"kubernetes.io/projected/8efb9f1a-3f05-4ddd-9ab5-7086f717524c-kube-api-access-b6j4m\") pod \"payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4\" (UID: \"8efb9f1a-3f05-4ddd-9ab5-7086f717524c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"
Oct 30 19:52:42.401638 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:42.401600    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"
Oct 30 19:52:42.719356 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:42.719327    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"]
Oct 30 19:52:42.720776 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:52:42.720740    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efb9f1a_3f05_4ddd_9ab5_7086f717524c.slice/crio-446f1d0fbb63d05ce52e3629e6118ef9f6121d07b76fe9182cfd70ec052442c6 WatchSource:0}: Error finding container 446f1d0fbb63d05ce52e3629e6118ef9f6121d07b76fe9182cfd70ec052442c6: Status 404 returned error can't find the container with id 446f1d0fbb63d05ce52e3629e6118ef9f6121d07b76fe9182cfd70ec052442c6
Oct 30 19:52:43.699674 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:43.699640    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4" event={"ID":"8efb9f1a-3f05-4ddd-9ab5-7086f717524c","Type":"ContainerStarted","Data":"ca4b71edc8b069e69dd77412443b79801f3e86d857374eddb2a65825ec8f1151"}
Oct 30 19:52:43.699849 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:43.699688    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4" event={"ID":"8efb9f1a-3f05-4ddd-9ab5-7086f717524c","Type":"ContainerStarted","Data":"446f1d0fbb63d05ce52e3629e6118ef9f6121d07b76fe9182cfd70ec052442c6"}
Oct 30 19:52:43.699849 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:43.699836    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"
Oct 30 19:52:43.723201 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:43.723146    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4" podStartSLOduration=1.723131575 podStartE2EDuration="1.723131575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:43.721402183 +0000 UTC m=+1242.637009039" watchObservedRunningTime="2026-10-30 19:52:43.723131575 +0000 UTC m=+1242.638738430"
Oct 30 19:52:47.697438 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:47.697408    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:52:54.310471 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:54.310438    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"]
Oct 30 19:52:54.314348 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:54.314329    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:52:54.323497 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:54.323472    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-426bab-serving-cert\""
Oct 30 19:52:54.329147 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:54.329119    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"]
Oct 30 19:52:54.454397 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:54.450794    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59ec827-e83b-4e1f-b138-bccae9cc5356-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2\" (UID: \"b59ec827-e83b-4e1f-b138-bccae9cc5356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:52:54.454397 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:54.450992    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59ec827-e83b-4e1f-b138-bccae9cc5356-metrics-certs\") pod \"maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2\" (UID: \"b59ec827-e83b-4e1f-b138-bccae9cc5356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:52:54.454397 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:54.451076    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxw2c\" (UniqueName: \"kubernetes.io/projected/b59ec827-e83b-4e1f-b138-bccae9cc5356-kube-api-access-lxw2c\") pod \"maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2\" (UID: \"b59ec827-e83b-4e1f-b138-bccae9cc5356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:52:54.454397 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:54.451119    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59ec827-e83b-4e1f-b138-bccae9cc5356-maas-api-tls\") pod \"maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2\" (UID: \"b59ec827-e83b-4e1f-b138-bccae9cc5356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:52:54.552769 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:54.552735    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59ec827-e83b-4e1f-b138-bccae9cc5356-metrics-certs\") pod \"maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2\" (UID: \"b59ec827-e83b-4e1f-b138-bccae9cc5356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:52:54.552961 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:54.552803    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxw2c\" (UniqueName: \"kubernetes.io/projected/b59ec827-e83b-4e1f-b138-bccae9cc5356-kube-api-access-lxw2c\") pod \"maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2\" (UID: \"b59ec827-e83b-4e1f-b138-bccae9cc5356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:52:54.552961 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:54.552844    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59ec827-e83b-4e1f-b138-bccae9cc5356-maas-api-tls\") pod \"maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2\" (UID: \"b59ec827-e83b-4e1f-b138-bccae9cc5356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:52:54.553069 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:52:54.552978    6030 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-serving-cert: secret "maas-api-e2e-ipp-a-426bab-serving-cert" not found
Oct 30 19:52:54.553069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:54.553021    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59ec827-e83b-4e1f-b138-bccae9cc5356-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2\" (UID: \"b59ec827-e83b-4e1f-b138-bccae9cc5356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:52:54.553159 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:52:54.553077    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b59ec827-e83b-4e1f-b138-bccae9cc5356-maas-api-tls podName:b59ec827-e83b-4e1f-b138-bccae9cc5356 nodeName:}" failed. No retries permitted until 2026-10-30 19:52:55.053022814 +0000 UTC m=+1253.968629656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b59ec827-e83b-4e1f-b138-bccae9cc5356-maas-api-tls") pod "maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2" (UID: "b59ec827-e83b-4e1f-b138-bccae9cc5356") : secret "maas-api-e2e-ipp-a-426bab-serving-cert" not found
Oct 30 19:52:54.553612 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:54.553595    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59ec827-e83b-4e1f-b138-bccae9cc5356-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2\" (UID: \"b59ec827-e83b-4e1f-b138-bccae9cc5356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:52:54.555438 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:54.555417    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59ec827-e83b-4e1f-b138-bccae9cc5356-metrics-certs\") pod \"maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2\" (UID: \"b59ec827-e83b-4e1f-b138-bccae9cc5356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:52:54.570254 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:54.570194    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxw2c\" (UniqueName: \"kubernetes.io/projected/b59ec827-e83b-4e1f-b138-bccae9cc5356-kube-api-access-lxw2c\") pod \"maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2\" (UID: \"b59ec827-e83b-4e1f-b138-bccae9cc5356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:52:54.710887 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:54.710859    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"
Oct 30 19:52:55.061826 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:55.061670    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59ec827-e83b-4e1f-b138-bccae9cc5356-maas-api-tls\") pod \"maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2\" (UID: \"b59ec827-e83b-4e1f-b138-bccae9cc5356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:52:55.071108 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:55.070339    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59ec827-e83b-4e1f-b138-bccae9cc5356-maas-api-tls\") pod \"maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2\" (UID: \"b59ec827-e83b-4e1f-b138-bccae9cc5356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:52:55.234087 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:55.232769    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:52:55.595510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:55.595480    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"]
Oct 30 19:52:55.596555 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:52:55.596527    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59ec827_e83b_4e1f_b138_bccae9cc5356.slice/crio-fe9d405c2a9e0a41342d371f7560b96de1ff6a07530eb9b443b4373088a5c621 WatchSource:0}: Error finding container fe9d405c2a9e0a41342d371f7560b96de1ff6a07530eb9b443b4373088a5c621: Status 404 returned error can't find the container with id fe9d405c2a9e0a41342d371f7560b96de1ff6a07530eb9b443b4373088a5c621
Oct 30 19:52:56.000784 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:56.000748    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2" event={"ID":"b59ec827-e83b-4e1f-b138-bccae9cc5356","Type":"ContainerStarted","Data":"fe9d405c2a9e0a41342d371f7560b96de1ff6a07530eb9b443b4373088a5c621"}
Oct 30 19:52:57.009575 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:57.009538    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2" event={"ID":"b59ec827-e83b-4e1f-b138-bccae9cc5356","Type":"ContainerStarted","Data":"412b579e110c38596854bdf12e94e9c35b6ef4fb5cacb8c31ea09e36f9eb5983"}
Oct 30 19:52:57.010025 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:52:57.009705    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:53:01.059210 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:01.059153    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2" podStartSLOduration=6.645081395 podStartE2EDuration="7.059138335s" totalImagesPullingTime="414.05694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:57.065926659 +0000 UTC m=+1255.981533515" watchObservedRunningTime="2026-10-30 19:53:01.059138335 +0000 UTC m=+1259.974745248"
Oct 30 19:53:01.060810 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:01.060790    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"]
Oct 30 19:53:01.061019 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:01.060994    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt" podUID="3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" containerName="istio-proxy" containerID="cri-o://0d28a11e3f821105935c0313cc69ae83e8d43578d6b7bb86c66fd22fcbb6fc6d" gracePeriod=30
Oct 30 19:53:02.024409 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:02.024378    6030 scope.go:122] "RemoveContainer" containerID="809973c7065d7cad9f9acbc6f6ebb8218a13b360559aba56c92688f992f9eaa0"
Oct 30 19:53:03.022566 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:03.022533    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:53:04.374520 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:53:04.374475    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:53:06.373408 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.373384    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:53:06.420279 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.420240    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-data\") pod \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") "
Oct 30 19:53:06.420472 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.420300    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-credential-socket\") pod \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") "
Oct 30 19:53:06.420472 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.420338    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istiod-ca-cert\") pod \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") "
Oct 30 19:53:06.420472 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.420378    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-token\" (UniqueName: \"kubernetes.io/projected/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-token\") pod \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") "
Oct 30 19:53:06.420658 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.420476    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-workload-socket\") pod \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") "
Oct 30 19:53:06.420658 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.420509    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-podinfo\") pod \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") "
Oct 30 19:53:06.420658 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.420550    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-workload-certs\") pod \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") "
Oct 30 19:53:06.420658 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.420640    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-kube-api-access-nf4j9\" (UniqueName: \"kubernetes.io/projected/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-kube-api-access-nf4j9\") pod \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") "
Oct 30 19:53:06.420863 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.420734    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istiod-ca-cert" pod "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" (UID: "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:06.420863 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.420758    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-envoy\") pod \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\" (UID: \"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd\") "
Oct 30 19:53:06.421296 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.421269    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-credential-socket" pod "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" (UID: "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:06.421480 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.421464    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:06.421563 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.421485    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:06.421672 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.421652    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-workload-certs" pod "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" (UID: "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:06.422145 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.422089    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-workload-socket" pod "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" (UID: "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:06.422278 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.422254    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-data" pod "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" (UID: "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:06.423518 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.423489    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-podinfo" pod "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" (UID: "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:53:06.425613 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.425491    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-token" pod "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" (UID: "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:06.425613 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.425588    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-kube-api-access-nf4j9" pod "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" (UID: "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd"). InnerVolumeSpecName "kube-api-access-nf4j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:06.431667 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.431645    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-envoy" pod "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" (UID: "3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:06.522241 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.522197    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:06.522241 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.522230    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:06.522241 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.522245    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:06.522475 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.522258    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf4j9\" (UniqueName: \"kubernetes.io/projected/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-kube-api-access-nf4j9\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:06.522475 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.522270    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:06.522475 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.522282    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:06.522475 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:06.522293    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:07.125016 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:07.124843    6030 generic.go:356] "Generic (PLEG): container finished" podID="3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" containerID="0d28a11e3f821105935c0313cc69ae83e8d43578d6b7bb86c66fd22fcbb6fc6d" exitCode=0
Oct 30 19:53:07.125479 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:07.125446    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt" event={"ID":"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd","Type":"ContainerDied","Data":"0d28a11e3f821105935c0313cc69ae83e8d43578d6b7bb86c66fd22fcbb6fc6d"}
Oct 30 19:53:07.125778 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:07.125746    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt" event={"ID":"3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd","Type":"ContainerDied","Data":"c774be4f3ca57522148562a5aa5998a17b2343162ee84082ea9a166e643ec00e"}
Oct 30 19:53:07.125932 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:07.125915    6030 scope.go:122] "RemoveContainer" containerID="0d28a11e3f821105935c0313cc69ae83e8d43578d6b7bb86c66fd22fcbb6fc6d"
Oct 30 19:53:07.126385 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:07.126366    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"
Oct 30 19:53:07.149340 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:07.149312    6030 scope.go:122] "RemoveContainer" containerID="0d28a11e3f821105935c0313cc69ae83e8d43578d6b7bb86c66fd22fcbb6fc6d"
Oct 30 19:53:07.149764 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:53:07.149739    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d28a11e3f821105935c0313cc69ae83e8d43578d6b7bb86c66fd22fcbb6fc6d\": container with ID starting with 0d28a11e3f821105935c0313cc69ae83e8d43578d6b7bb86c66fd22fcbb6fc6d not found: ID does not exist" containerID="0d28a11e3f821105935c0313cc69ae83e8d43578d6b7bb86c66fd22fcbb6fc6d"
Oct 30 19:53:07.149863 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:07.149772    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d28a11e3f821105935c0313cc69ae83e8d43578d6b7bb86c66fd22fcbb6fc6d"} err="failed to get container status \"0d28a11e3f821105935c0313cc69ae83e8d43578d6b7bb86c66fd22fcbb6fc6d\": rpc error: code = NotFound desc = could not find container \"0d28a11e3f821105935c0313cc69ae83e8d43578d6b7bb86c66fd22fcbb6fc6d\": container with ID starting with 0d28a11e3f821105935c0313cc69ae83e8d43578d6b7bb86c66fd22fcbb6fc6d not found: ID does not exist"
Oct 30 19:53:07.166495 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:07.166465    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"]
Oct 30 19:53:07.172609 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:07.172581    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-pd9zt"]
Oct 30 19:53:07.201304 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:07.201279    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" path="/var/lib/kubelet/pods/3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd/volumes"
Oct 30 19:53:07.969263 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:07.968262    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"]
Oct 30 19:53:07.969687 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:07.969375    6030 cpu_manager.go:414] "containerMap: removing container" podUID="3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" containerName="istio-proxy"
Oct 30 19:53:07.969687 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:07.969392    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" containerName="istio-proxy"
Oct 30 19:53:07.969687 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:07.969526    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a121fcd-2a6f-4d1e-9625-d8ccf6868cbd" containerName="istio-proxy"
Oct 30 19:53:07.974604 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:07.974583    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:53:07.981624 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:07.981604    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-ee4af9-serving-cert\""
Oct 30 19:53:07.992524 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:07.992499    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"]
Oct 30 19:53:08.042770 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:08.042732    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e94be7d-8ae8-4249-9270-691667f8b406-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp\" (UID: \"5e94be7d-8ae8-4249-9270-691667f8b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:53:08.042933 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:08.042789    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwdpb\" (UniqueName: \"kubernetes.io/projected/5e94be7d-8ae8-4249-9270-691667f8b406-kube-api-access-nwdpb\") pod \"maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp\" (UID: \"5e94be7d-8ae8-4249-9270-691667f8b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:53:08.042933 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:08.042898    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e94be7d-8ae8-4249-9270-691667f8b406-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp\" (UID: \"5e94be7d-8ae8-4249-9270-691667f8b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:53:08.043133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:08.043117    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e94be7d-8ae8-4249-9270-691667f8b406-metrics-certs\") pod \"maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp\" (UID: \"5e94be7d-8ae8-4249-9270-691667f8b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:53:08.170079 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:08.167597    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e94be7d-8ae8-4249-9270-691667f8b406-metrics-certs\") pod \"maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp\" (UID: \"5e94be7d-8ae8-4249-9270-691667f8b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:53:08.170079 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:08.167780    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e94be7d-8ae8-4249-9270-691667f8b406-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp\" (UID: \"5e94be7d-8ae8-4249-9270-691667f8b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:53:08.170079 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:08.167834    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwdpb\" (UniqueName: \"kubernetes.io/projected/5e94be7d-8ae8-4249-9270-691667f8b406-kube-api-access-nwdpb\") pod \"maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp\" (UID: \"5e94be7d-8ae8-4249-9270-691667f8b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:53:08.170079 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:08.167872    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e94be7d-8ae8-4249-9270-691667f8b406-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp\" (UID: \"5e94be7d-8ae8-4249-9270-691667f8b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:53:08.170079 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:53:08.168184    6030 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-serving-cert: secret "maas-api-e2e-praxis-b-ee4af9-serving-cert" not found
Oct 30 19:53:08.170079 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:53:08.168253    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e94be7d-8ae8-4249-9270-691667f8b406-maas-api-tls podName:5e94be7d-8ae8-4249-9270-691667f8b406 nodeName:}" failed. No retries permitted until 2026-10-30 19:53:08.668233702 +0000 UTC m=+1267.583840538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5e94be7d-8ae8-4249-9270-691667f8b406-maas-api-tls") pod "maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp" (UID: "5e94be7d-8ae8-4249-9270-691667f8b406") : secret "maas-api-e2e-praxis-b-ee4af9-serving-cert" not found
Oct 30 19:53:08.173076 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:08.172710    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e94be7d-8ae8-4249-9270-691667f8b406-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp\" (UID: \"5e94be7d-8ae8-4249-9270-691667f8b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:53:08.180234 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:08.179760    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e94be7d-8ae8-4249-9270-691667f8b406-metrics-certs\") pod \"maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp\" (UID: \"5e94be7d-8ae8-4249-9270-691667f8b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:53:08.195641 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:08.192237    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwdpb\" (UniqueName: \"kubernetes.io/projected/5e94be7d-8ae8-4249-9270-691667f8b406-kube-api-access-nwdpb\") pod \"maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp\" (UID: \"5e94be7d-8ae8-4249-9270-691667f8b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:53:08.674994 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:08.674952    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e94be7d-8ae8-4249-9270-691667f8b406-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp\" (UID: \"5e94be7d-8ae8-4249-9270-691667f8b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:53:08.677991 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:08.677966    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e94be7d-8ae8-4249-9270-691667f8b406-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp\" (UID: \"5e94be7d-8ae8-4249-9270-691667f8b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:53:08.898066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:08.894280    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:53:09.320568 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:09.319700    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"]
Oct 30 19:53:09.320568 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:09.320449    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6" podUID="842afcb7-9473-4e5a-b9fa-3486a236c1a5" containerName="istio-proxy" containerID="cri-o://06c0f26176e3bcdb99c82bea78d7e361b7620f4c663923eb2beb3ba344b06ad0" gracePeriod=30
Oct 30 19:53:09.341448 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:09.339560    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"]
Oct 30 19:53:10.256925 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:10.256853    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp" event={"ID":"5e94be7d-8ae8-4249-9270-691667f8b406","Type":"ContainerStarted","Data":"3e85e704c4051d54abad217cdc99b9b77c62a88e655001b254476d3a8d44938b"}
Oct 30 19:53:10.257153 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:10.256983    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp" event={"ID":"5e94be7d-8ae8-4249-9270-691667f8b406","Type":"ContainerStarted","Data":"73e422ba56befd939066755575520ac7d08c2d9963b182ef7890a1e1175f2b08"}
Oct 30 19:53:10.257153 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:10.257014    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:53:10.295794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:10.295715    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp" podStartSLOduration=2.801209667 podStartE2EDuration="3.295700989s" totalImagesPullingTime="494.491322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:10.292593574 +0000 UTC m=+1269.208200431" watchObservedRunningTime="2026-10-30 19:53:10.295700989 +0000 UTC m=+1269.211307844"
Oct 30 19:53:14.608454 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.608387    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:53:14.700366 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.700334    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-podinfo\") pod \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") "
Oct 30 19:53:14.700558 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.700419    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-envoy\") pod \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") "
Oct 30 19:53:14.700558 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.700452    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842afcb7-9473-4e5a-b9fa-3486a236c1a5-kube-api-access-hvht4\" (UniqueName: \"kubernetes.io/projected/842afcb7-9473-4e5a-b9fa-3486a236c1a5-kube-api-access-hvht4\") pod \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") "
Oct 30 19:53:14.700558 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.700477    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-token\" (UniqueName: \"kubernetes.io/projected/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-token\") pod \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") "
Oct 30 19:53:14.700558 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.700521    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-credential-socket\") pod \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") "
Oct 30 19:53:14.700806 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.700558    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istiod-ca-cert\") pod \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") "
Oct 30 19:53:14.700806 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.700713    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-data\") pod \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") "
Oct 30 19:53:14.701135 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.700857    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-workload-socket\") pod \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") "
Oct 30 19:53:14.701135 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.700902    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-workload-certs\") pod \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\" (UID: \"842afcb7-9473-4e5a-b9fa-3486a236c1a5\") "
Oct 30 19:53:14.701275 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.701085    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-data" pod "842afcb7-9473-4e5a-b9fa-3486a236c1a5" (UID: "842afcb7-9473-4e5a-b9fa-3486a236c1a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:14.701641 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.701616    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:14.701836 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.701805    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-workload-certs" pod "842afcb7-9473-4e5a-b9fa-3486a236c1a5" (UID: "842afcb7-9473-4e5a-b9fa-3486a236c1a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:14.701961 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.701847    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-workload-socket" pod "842afcb7-9473-4e5a-b9fa-3486a236c1a5" (UID: "842afcb7-9473-4e5a-b9fa-3486a236c1a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:14.702766 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.702709    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-credential-socket" pod "842afcb7-9473-4e5a-b9fa-3486a236c1a5" (UID: "842afcb7-9473-4e5a-b9fa-3486a236c1a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:14.703626 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.703586    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istiod-ca-cert" pod "842afcb7-9473-4e5a-b9fa-3486a236c1a5" (UID: "842afcb7-9473-4e5a-b9fa-3486a236c1a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:14.705154 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.705126    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-envoy" pod "842afcb7-9473-4e5a-b9fa-3486a236c1a5" (UID: "842afcb7-9473-4e5a-b9fa-3486a236c1a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:14.705262 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.705199    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-podinfo" pod "842afcb7-9473-4e5a-b9fa-3486a236c1a5" (UID: "842afcb7-9473-4e5a-b9fa-3486a236c1a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:53:14.705335 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.705323    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-token" pod "842afcb7-9473-4e5a-b9fa-3486a236c1a5" (UID: "842afcb7-9473-4e5a-b9fa-3486a236c1a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:14.705819 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.705793    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842afcb7-9473-4e5a-b9fa-3486a236c1a5-kube-api-access-hvht4" pod "842afcb7-9473-4e5a-b9fa-3486a236c1a5" (UID: "842afcb7-9473-4e5a-b9fa-3486a236c1a5"). InnerVolumeSpecName "kube-api-access-hvht4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:14.803244 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.803206    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:14.803244 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.803237    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvht4\" (UniqueName: \"kubernetes.io/projected/842afcb7-9473-4e5a-b9fa-3486a236c1a5-kube-api-access-hvht4\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:14.803455 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.803253    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:14.803455 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.803288    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:14.803455 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.803298    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:14.803455 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.803308    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:14.803455 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.803316    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842afcb7-9473-4e5a-b9fa-3486a236c1a5-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:14.803455 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:14.803328    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842afcb7-9473-4e5a-b9fa-3486a236c1a5-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:15.288850 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:15.288818    6030 generic.go:356] "Generic (PLEG): container finished" podID="842afcb7-9473-4e5a-b9fa-3486a236c1a5" containerID="06c0f26176e3bcdb99c82bea78d7e361b7620f4c663923eb2beb3ba344b06ad0" exitCode=0
Oct 30 19:53:15.289014 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:15.288912    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"
Oct 30 19:53:15.289014 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:15.288947    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6" event={"ID":"842afcb7-9473-4e5a-b9fa-3486a236c1a5","Type":"ContainerDied","Data":"06c0f26176e3bcdb99c82bea78d7e361b7620f4c663923eb2beb3ba344b06ad0"}
Oct 30 19:53:15.289014 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:15.288999    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6" event={"ID":"842afcb7-9473-4e5a-b9fa-3486a236c1a5","Type":"ContainerDied","Data":"1b28129fa14aedacfc205c0d2078950d41198d897d8caa5de476005bca4ab117"}
Oct 30 19:53:15.289150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:15.289020    6030 scope.go:122] "RemoveContainer" containerID="06c0f26176e3bcdb99c82bea78d7e361b7620f4c663923eb2beb3ba344b06ad0"
Oct 30 19:53:15.299230 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:15.299201    6030 scope.go:122] "RemoveContainer" containerID="06c0f26176e3bcdb99c82bea78d7e361b7620f4c663923eb2beb3ba344b06ad0"
Oct 30 19:53:15.299514 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:53:15.299495    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c0f26176e3bcdb99c82bea78d7e361b7620f4c663923eb2beb3ba344b06ad0\": container with ID starting with 06c0f26176e3bcdb99c82bea78d7e361b7620f4c663923eb2beb3ba344b06ad0 not found: ID does not exist" containerID="06c0f26176e3bcdb99c82bea78d7e361b7620f4c663923eb2beb3ba344b06ad0"
Oct 30 19:53:15.299597 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:15.299522    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c0f26176e3bcdb99c82bea78d7e361b7620f4c663923eb2beb3ba344b06ad0"} err="failed to get container status \"06c0f26176e3bcdb99c82bea78d7e361b7620f4c663923eb2beb3ba344b06ad0\": rpc error: code = NotFound desc = could not find container \"06c0f26176e3bcdb99c82bea78d7e361b7620f4c663923eb2beb3ba344b06ad0\": container with ID starting with 06c0f26176e3bcdb99c82bea78d7e361b7620f4c663923eb2beb3ba344b06ad0 not found: ID does not exist"
Oct 30 19:53:15.314517 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:15.314494    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"]
Oct 30 19:53:15.323069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:15.323022    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-hw4t6"]
Oct 30 19:53:15.945503 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:15.945471    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"]
Oct 30 19:53:16.268971 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:16.268894    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:53:17.201697 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:17.201661    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842afcb7-9473-4e5a-b9fa-3486a236c1a5" path="/var/lib/kubelet/pods/842afcb7-9473-4e5a-b9fa-3486a236c1a5/volumes"
Oct 30 19:53:20.949318 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:20.948073    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4" podUID="8efb9f1a-3f05-4ddd-9ab5-7086f717524c" containerName="payload-pre-processing" containerID="cri-o://ca4b71edc8b069e69dd77412443b79801f3e86d857374eddb2a65825ec8f1151" gracePeriod=25
Oct 30 19:53:21.267379 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:21.267352    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"
Oct 30 19:53:21.344425 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:21.344396    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8efb9f1a-3f05-4ddd-9ab5-7086f717524c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efb9f1a-3f05-4ddd-9ab5-7086f717524c-plugins-config-volume\") pod \"8efb9f1a-3f05-4ddd-9ab5-7086f717524c\" (UID: \"8efb9f1a-3f05-4ddd-9ab5-7086f717524c\") "
Oct 30 19:53:21.344583 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:21.344455    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8efb9f1a-3f05-4ddd-9ab5-7086f717524c-kube-api-access-b6j4m\" (UniqueName: \"kubernetes.io/projected/8efb9f1a-3f05-4ddd-9ab5-7086f717524c-kube-api-access-b6j4m\") pod \"8efb9f1a-3f05-4ddd-9ab5-7086f717524c\" (UID: \"8efb9f1a-3f05-4ddd-9ab5-7086f717524c\") "
Oct 30 19:53:21.344794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:21.344768    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8efb9f1a-3f05-4ddd-9ab5-7086f717524c-plugins-config-volume" pod "8efb9f1a-3f05-4ddd-9ab5-7086f717524c" (UID: "8efb9f1a-3f05-4ddd-9ab5-7086f717524c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:21.345068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:21.345033    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efb9f1a-3f05-4ddd-9ab5-7086f717524c-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:21.346689 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:21.346665    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efb9f1a-3f05-4ddd-9ab5-7086f717524c-kube-api-access-b6j4m" pod "8efb9f1a-3f05-4ddd-9ab5-7086f717524c" (UID: "8efb9f1a-3f05-4ddd-9ab5-7086f717524c"). InnerVolumeSpecName "kube-api-access-b6j4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:21.373678 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:21.373637    6030 generic.go:356] "Generic (PLEG): container finished" podID="8efb9f1a-3f05-4ddd-9ab5-7086f717524c" containerID="ca4b71edc8b069e69dd77412443b79801f3e86d857374eddb2a65825ec8f1151" exitCode=0
Oct 30 19:53:21.373864 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:21.373690    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4" event={"ID":"8efb9f1a-3f05-4ddd-9ab5-7086f717524c","Type":"ContainerDied","Data":"ca4b71edc8b069e69dd77412443b79801f3e86d857374eddb2a65825ec8f1151"}
Oct 30 19:53:21.373864 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:21.373745    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4" event={"ID":"8efb9f1a-3f05-4ddd-9ab5-7086f717524c","Type":"ContainerDied","Data":"446f1d0fbb63d05ce52e3629e6118ef9f6121d07b76fe9182cfd70ec052442c6"}
Oct 30 19:53:21.373864 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:21.373766    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"
Oct 30 19:53:21.373864 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:21.373778    6030 scope.go:122] "RemoveContainer" containerID="ca4b71edc8b069e69dd77412443b79801f3e86d857374eddb2a65825ec8f1151"
Oct 30 19:53:21.385794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:21.385768    6030 scope.go:122] "RemoveContainer" containerID="ca4b71edc8b069e69dd77412443b79801f3e86d857374eddb2a65825ec8f1151"
Oct 30 19:53:21.386073 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:53:21.386025    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca4b71edc8b069e69dd77412443b79801f3e86d857374eddb2a65825ec8f1151\": container with ID starting with ca4b71edc8b069e69dd77412443b79801f3e86d857374eddb2a65825ec8f1151 not found: ID does not exist" containerID="ca4b71edc8b069e69dd77412443b79801f3e86d857374eddb2a65825ec8f1151"
Oct 30 19:53:21.386159 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:21.386072    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4b71edc8b069e69dd77412443b79801f3e86d857374eddb2a65825ec8f1151"} err="failed to get container status \"ca4b71edc8b069e69dd77412443b79801f3e86d857374eddb2a65825ec8f1151\": rpc error: code = NotFound desc = could not find container \"ca4b71edc8b069e69dd77412443b79801f3e86d857374eddb2a65825ec8f1151\": container with ID starting with ca4b71edc8b069e69dd77412443b79801f3e86d857374eddb2a65825ec8f1151 not found: ID does not exist"
Oct 30 19:53:21.401649 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:21.401617    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"]
Oct 30 19:53:21.404072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:21.404015    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-659f447667-7fkg4"]
Oct 30 19:53:21.446412 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:21.446383    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6j4m\" (UniqueName: \"kubernetes.io/projected/8efb9f1a-3f05-4ddd-9ab5-7086f717524c-kube-api-access-b6j4m\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:23.229621 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:53:23.229591    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8efb9f1a-3f05-4ddd-9ab5-7086f717524c" path="/var/lib/kubelet/pods/8efb9f1a-3f05-4ddd-9ab5-7086f717524c/volumes"
Oct 30 19:53:34.288887 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:53:34.288844    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:54:00.710877 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:00.710807    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"]
Oct 30 19:54:00.713150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:00.712216    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz" podUID="cac65540-89ac-4591-b288-14f8ac45be99" containerName="maas-api" containerID="cri-o://b49b23d80fcff213d355968360d120a841a5569d899a72198026ae1c9c79a114" gracePeriod=30
Oct 30 19:54:00.799075 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:00.799008    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"]
Oct 30 19:54:00.799312 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:00.799266    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc" podUID="8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5" containerName="payload-processing" containerID="cri-o://32e5a408bfc272d8ffdaa9d479e6ba2a96f3bc17539c2b5c9455829cdeeef408" gracePeriod=30
Oct 30 19:54:00.809801 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:00.809749    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"]
Oct 30 19:54:00.810351 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:00.810317    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8" podUID="14294872-1aaf-4b7d-b2ab-f80fbf7db165" containerName="payload-pre-processing" containerID="cri-o://66a141525b994f847211674c755ebcec6b1ee15cbff7c2df48833479eeb753ee" gracePeriod=30
Oct 30 19:54:00.947552 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:54:00.947506    6030 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14294872_1aaf_4b7d_b2ab_f80fbf7db165.slice/crio-conmon-66a141525b994f847211674c755ebcec6b1ee15cbff7c2df48833479eeb753ee.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:54:01.032593 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.032496    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz" podUID="cac65540-89ac-4591-b288-14f8ac45be99" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.95:8443/health\": dial tcp 10.133.0.95:8443: connect: connection refused"
Oct 30 19:54:01.358599 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.358545    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:54:01.430399 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.430370    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"
Oct 30 19:54:01.441573 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.441530    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"
Oct 30 19:54:01.506503 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.506464    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5-kube-api-access-rkrjz\" (UniqueName: \"kubernetes.io/projected/8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5-kube-api-access-rkrjz\") pod \"8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5\" (UID: \"8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5\") "
Oct 30 19:54:01.506694 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.506522    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5-plugins-config-volume\") pod \"8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5\" (UID: \"8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5\") "
Oct 30 19:54:01.506694 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.506630    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac65540-89ac-4591-b288-14f8ac45be99-kube-api-access-5n7xn\" (UniqueName: \"kubernetes.io/projected/cac65540-89ac-4591-b288-14f8ac45be99-kube-api-access-5n7xn\") pod \"cac65540-89ac-4591-b288-14f8ac45be99\" (UID: \"cac65540-89ac-4591-b288-14f8ac45be99\") "
Oct 30 19:54:01.506830 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.506783    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cac65540-89ac-4591-b288-14f8ac45be99-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac65540-89ac-4591-b288-14f8ac45be99-metrics-certs\") pod \"cac65540-89ac-4591-b288-14f8ac45be99\" (UID: \"cac65540-89ac-4591-b288-14f8ac45be99\") "
Oct 30 19:54:01.506884 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.506838    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cac65540-89ac-4591-b288-14f8ac45be99-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac65540-89ac-4591-b288-14f8ac45be99-maas-api-tls\") pod \"cac65540-89ac-4591-b288-14f8ac45be99\" (UID: \"cac65540-89ac-4591-b288-14f8ac45be99\") "
Oct 30 19:54:01.527970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.506881    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14294872-1aaf-4b7d-b2ab-f80fbf7db165-kube-api-access-vv42h\" (UniqueName: \"kubernetes.io/projected/14294872-1aaf-4b7d-b2ab-f80fbf7db165-kube-api-access-vv42h\") pod \"14294872-1aaf-4b7d-b2ab-f80fbf7db165\" (UID: \"14294872-1aaf-4b7d-b2ab-f80fbf7db165\") "
Oct 30 19:54:01.527970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.506979    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cac65540-89ac-4591-b288-14f8ac45be99-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac65540-89ac-4591-b288-14f8ac45be99-openshift-service-ca\") pod \"cac65540-89ac-4591-b288-14f8ac45be99\" (UID: \"cac65540-89ac-4591-b288-14f8ac45be99\") "
Oct 30 19:54:01.527970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.507018    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14294872-1aaf-4b7d-b2ab-f80fbf7db165-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14294872-1aaf-4b7d-b2ab-f80fbf7db165-plugins-config-volume\") pod \"14294872-1aaf-4b7d-b2ab-f80fbf7db165\" (UID: \"14294872-1aaf-4b7d-b2ab-f80fbf7db165\") "
Oct 30 19:54:01.527970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.508189    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac65540-89ac-4591-b288-14f8ac45be99-openshift-service-ca" pod "cac65540-89ac-4591-b288-14f8ac45be99" (UID: "cac65540-89ac-4591-b288-14f8ac45be99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:01.527970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.508960    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14294872-1aaf-4b7d-b2ab-f80fbf7db165-plugins-config-volume" pod "14294872-1aaf-4b7d-b2ab-f80fbf7db165" (UID: "14294872-1aaf-4b7d-b2ab-f80fbf7db165"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:01.527970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.509565    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5-plugins-config-volume" pod "8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5" (UID: "8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:01.527970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.514482    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac65540-89ac-4591-b288-14f8ac45be99-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:01.527970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.514600    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14294872-1aaf-4b7d-b2ab-f80fbf7db165-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:01.527970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.514624    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:01.579301 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.579191    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac65540-89ac-4591-b288-14f8ac45be99-metrics-certs" pod "cac65540-89ac-4591-b288-14f8ac45be99" (UID: "cac65540-89ac-4591-b288-14f8ac45be99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:01.580033 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.579955    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5-kube-api-access-rkrjz" pod "8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5" (UID: "8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5"). InnerVolumeSpecName "kube-api-access-rkrjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:01.580033 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.579976    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac65540-89ac-4591-b288-14f8ac45be99-kube-api-access-5n7xn" pod "cac65540-89ac-4591-b288-14f8ac45be99" (UID: "cac65540-89ac-4591-b288-14f8ac45be99"). InnerVolumeSpecName "kube-api-access-5n7xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:01.580578 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.580554    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14294872-1aaf-4b7d-b2ab-f80fbf7db165-kube-api-access-vv42h" pod "14294872-1aaf-4b7d-b2ab-f80fbf7db165" (UID: "14294872-1aaf-4b7d-b2ab-f80fbf7db165"). InnerVolumeSpecName "kube-api-access-vv42h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:01.581670 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.581644    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac65540-89ac-4591-b288-14f8ac45be99-maas-api-tls" pod "cac65540-89ac-4591-b288-14f8ac45be99" (UID: "cac65540-89ac-4591-b288-14f8ac45be99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:01.616467 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.616442    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkrjz\" (UniqueName: \"kubernetes.io/projected/8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5-kube-api-access-rkrjz\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:01.616467 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.616465    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n7xn\" (UniqueName: \"kubernetes.io/projected/cac65540-89ac-4591-b288-14f8ac45be99-kube-api-access-5n7xn\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:01.616633 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.616477    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac65540-89ac-4591-b288-14f8ac45be99-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:01.616633 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.616488    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac65540-89ac-4591-b288-14f8ac45be99-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:01.616633 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.616499    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv42h\" (UniqueName: \"kubernetes.io/projected/14294872-1aaf-4b7d-b2ab-f80fbf7db165-kube-api-access-vv42h\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:01.756880 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.756843    6030 generic.go:356] "Generic (PLEG): container finished" podID="8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5" containerID="32e5a408bfc272d8ffdaa9d479e6ba2a96f3bc17539c2b5c9455829cdeeef408" exitCode=0
Oct 30 19:54:01.757379 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.756900    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"
Oct 30 19:54:01.757379 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.756927    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc" event={"ID":"8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5","Type":"ContainerDied","Data":"32e5a408bfc272d8ffdaa9d479e6ba2a96f3bc17539c2b5c9455829cdeeef408"}
Oct 30 19:54:01.757379 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.756982    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc" event={"ID":"8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5","Type":"ContainerDied","Data":"b29714bc2182ba8eeb8094f9e9d2049ef47e576f7ded61bffc74f0a8576df3e5"}
Oct 30 19:54:01.757379 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.757007    6030 scope.go:122] "RemoveContainer" containerID="32e5a408bfc272d8ffdaa9d479e6ba2a96f3bc17539c2b5c9455829cdeeef408"
Oct 30 19:54:01.758635 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.758611    6030 generic.go:356] "Generic (PLEG): container finished" podID="cac65540-89ac-4591-b288-14f8ac45be99" containerID="b49b23d80fcff213d355968360d120a841a5569d899a72198026ae1c9c79a114" exitCode=0
Oct 30 19:54:01.758776 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.758682    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"
Oct 30 19:54:01.758880 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.758851    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz" event={"ID":"cac65540-89ac-4591-b288-14f8ac45be99","Type":"ContainerDied","Data":"b49b23d80fcff213d355968360d120a841a5569d899a72198026ae1c9c79a114"}
Oct 30 19:54:01.758993 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.758925    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz" event={"ID":"cac65540-89ac-4591-b288-14f8ac45be99","Type":"ContainerDied","Data":"4790d64c396216c61ecafc384675ab34048b8658d4522b774e214ba8e5fe7a32"}
Oct 30 19:54:01.760326 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.760303    6030 generic.go:356] "Generic (PLEG): container finished" podID="14294872-1aaf-4b7d-b2ab-f80fbf7db165" containerID="66a141525b994f847211674c755ebcec6b1ee15cbff7c2df48833479eeb753ee" exitCode=0
Oct 30 19:54:01.760440 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.760347    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8" event={"ID":"14294872-1aaf-4b7d-b2ab-f80fbf7db165","Type":"ContainerDied","Data":"66a141525b994f847211674c755ebcec6b1ee15cbff7c2df48833479eeb753ee"}
Oct 30 19:54:01.760440 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.760360    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"
Oct 30 19:54:01.760440 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.760381    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8" event={"ID":"14294872-1aaf-4b7d-b2ab-f80fbf7db165","Type":"ContainerDied","Data":"6c2973231590bf988a8aa50084a0305c5f0a7d6f3e05503df926272958a6a5b1"}
Oct 30 19:54:01.768720 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.768699    6030 scope.go:122] "RemoveContainer" containerID="32e5a408bfc272d8ffdaa9d479e6ba2a96f3bc17539c2b5c9455829cdeeef408"
Oct 30 19:54:01.768996 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:54:01.768970    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e5a408bfc272d8ffdaa9d479e6ba2a96f3bc17539c2b5c9455829cdeeef408\": container with ID starting with 32e5a408bfc272d8ffdaa9d479e6ba2a96f3bc17539c2b5c9455829cdeeef408 not found: ID does not exist" containerID="32e5a408bfc272d8ffdaa9d479e6ba2a96f3bc17539c2b5c9455829cdeeef408"
Oct 30 19:54:01.769122 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.769002    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e5a408bfc272d8ffdaa9d479e6ba2a96f3bc17539c2b5c9455829cdeeef408"} err="failed to get container status \"32e5a408bfc272d8ffdaa9d479e6ba2a96f3bc17539c2b5c9455829cdeeef408\": rpc error: code = NotFound desc = could not find container \"32e5a408bfc272d8ffdaa9d479e6ba2a96f3bc17539c2b5c9455829cdeeef408\": container with ID starting with 32e5a408bfc272d8ffdaa9d479e6ba2a96f3bc17539c2b5c9455829cdeeef408 not found: ID does not exist"
Oct 30 19:54:01.769122 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.769017    6030 scope.go:122] "RemoveContainer" containerID="b49b23d80fcff213d355968360d120a841a5569d899a72198026ae1c9c79a114"
Oct 30 19:54:01.778636 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.778619    6030 scope.go:122] "RemoveContainer" containerID="b49b23d80fcff213d355968360d120a841a5569d899a72198026ae1c9c79a114"
Oct 30 19:54:01.778897 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:54:01.778880    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b49b23d80fcff213d355968360d120a841a5569d899a72198026ae1c9c79a114\": container with ID starting with b49b23d80fcff213d355968360d120a841a5569d899a72198026ae1c9c79a114 not found: ID does not exist" containerID="b49b23d80fcff213d355968360d120a841a5569d899a72198026ae1c9c79a114"
Oct 30 19:54:01.778968 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.778904    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b49b23d80fcff213d355968360d120a841a5569d899a72198026ae1c9c79a114"} err="failed to get container status \"b49b23d80fcff213d355968360d120a841a5569d899a72198026ae1c9c79a114\": rpc error: code = NotFound desc = could not find container \"b49b23d80fcff213d355968360d120a841a5569d899a72198026ae1c9c79a114\": container with ID starting with b49b23d80fcff213d355968360d120a841a5569d899a72198026ae1c9c79a114 not found: ID does not exist"
Oct 30 19:54:01.778968 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.778919    6030 scope.go:122] "RemoveContainer" containerID="66a141525b994f847211674c755ebcec6b1ee15cbff7c2df48833479eeb753ee"
Oct 30 19:54:01.787815 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.787794    6030 scope.go:122] "RemoveContainer" containerID="66a141525b994f847211674c755ebcec6b1ee15cbff7c2df48833479eeb753ee"
Oct 30 19:54:01.788134 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:54:01.788110    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a141525b994f847211674c755ebcec6b1ee15cbff7c2df48833479eeb753ee\": container with ID starting with 66a141525b994f847211674c755ebcec6b1ee15cbff7c2df48833479eeb753ee not found: ID does not exist" containerID="66a141525b994f847211674c755ebcec6b1ee15cbff7c2df48833479eeb753ee"
Oct 30 19:54:01.788220 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.788138    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a141525b994f847211674c755ebcec6b1ee15cbff7c2df48833479eeb753ee"} err="failed to get container status \"66a141525b994f847211674c755ebcec6b1ee15cbff7c2df48833479eeb753ee\": rpc error: code = NotFound desc = could not find container \"66a141525b994f847211674c755ebcec6b1ee15cbff7c2df48833479eeb753ee\": container with ID starting with 66a141525b994f847211674c755ebcec6b1ee15cbff7c2df48833479eeb753ee not found: ID does not exist"
Oct 30 19:54:01.788895 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.788875    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"]
Oct 30 19:54:01.793849 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.793825    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59e21a-7b48b6cc86-8r4tc"]
Oct 30 19:54:01.808305 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.808283    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"]
Oct 30 19:54:01.812652 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.812632    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59e21a-688fbf8b95-kqkp8"]
Oct 30 19:54:01.828393 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.828370    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"]
Oct 30 19:54:01.831266 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:01.831218    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-59e21a-7787f5d859-d9wgz"]
Oct 30 19:54:03.247071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:03.241815    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14294872-1aaf-4b7d-b2ab-f80fbf7db165" path="/var/lib/kubelet/pods/14294872-1aaf-4b7d-b2ab-f80fbf7db165/volumes"
Oct 30 19:54:03.247071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:03.242430    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5" path="/var/lib/kubelet/pods/8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5/volumes"
Oct 30 19:54:03.247071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:03.243014    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac65540-89ac-4591-b288-14f8ac45be99" path="/var/lib/kubelet/pods/cac65540-89ac-4591-b288-14f8ac45be99/volumes"
Oct 30 19:54:04.363922 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:54:04.363876    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:54:10.695995 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:10.695953    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"]
Oct 30 19:54:10.696430 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:10.696232    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj" podUID="8de060ca-04d8-4c9e-aff5-78d3dbbc0199" containerName="istio-proxy" containerID="cri-o://3f940f3d46e522c8876bedb87919cefab88c9e573f3af29248eeb2ebbdbd595c" gracePeriod=30
Oct 30 19:54:15.905544 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:15.905187    6030 generic.go:356] "Generic (PLEG): container finished" podID="8de060ca-04d8-4c9e-aff5-78d3dbbc0199" containerID="3f940f3d46e522c8876bedb87919cefab88c9e573f3af29248eeb2ebbdbd595c" exitCode=0
Oct 30 19:54:15.906351 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:15.906119    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj" event={"ID":"8de060ca-04d8-4c9e-aff5-78d3dbbc0199","Type":"ContainerDied","Data":"3f940f3d46e522c8876bedb87919cefab88c9e573f3af29248eeb2ebbdbd595c"}
Oct 30 19:54:15.906351 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:15.906205    6030 scope.go:122] "RemoveContainer" containerID="c580b2b203e5f0e9ac68df6aade37d1facda218fd802604e2ea2593f2732c2bc"
Oct 30 19:54:15.973239 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:15.973214    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:54:16.063778 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.063683    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-token\" (UniqueName: \"kubernetes.io/projected/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-token\") pod \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") "
Oct 30 19:54:16.063778 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.063741    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-kube-api-access-cdd5p\" (UniqueName: \"kubernetes.io/projected/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-kube-api-access-cdd5p\") pod \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") "
Oct 30 19:54:16.064006 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.063817    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-credential-socket\") pod \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") "
Oct 30 19:54:16.064006 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.063870    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istiod-ca-cert\") pod \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") "
Oct 30 19:54:16.064006 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.063918    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-workload-socket\") pod \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") "
Oct 30 19:54:16.064006 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.063959    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-podinfo\") pod \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") "
Oct 30 19:54:16.064006 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.063987    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-workload-certs\") pod \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") "
Oct 30 19:54:16.064303 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.064036    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-envoy\") pod \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") "
Oct 30 19:54:16.064303 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.064160    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-data\") pod \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\" (UID: \"8de060ca-04d8-4c9e-aff5-78d3dbbc0199\") "
Oct 30 19:54:16.064303 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.064251    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-credential-socket" pod "8de060ca-04d8-4c9e-aff5-78d3dbbc0199" (UID: "8de060ca-04d8-4c9e-aff5-78d3dbbc0199"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:16.064303 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.064279    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-workload-socket" pod "8de060ca-04d8-4c9e-aff5-78d3dbbc0199" (UID: "8de060ca-04d8-4c9e-aff5-78d3dbbc0199"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:16.064485 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.064309    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istiod-ca-cert" pod "8de060ca-04d8-4c9e-aff5-78d3dbbc0199" (UID: "8de060ca-04d8-4c9e-aff5-78d3dbbc0199"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:16.064540 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.064489    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-workload-certs" pod "8de060ca-04d8-4c9e-aff5-78d3dbbc0199" (UID: "8de060ca-04d8-4c9e-aff5-78d3dbbc0199"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:16.064764 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.064744    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-data" pod "8de060ca-04d8-4c9e-aff5-78d3dbbc0199" (UID: "8de060ca-04d8-4c9e-aff5-78d3dbbc0199"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:16.064822 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.064751    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:16.064822 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.064794    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:16.064822 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.064813    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:16.064913 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.064829    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:16.066155 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.066128    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-token" pod "8de060ca-04d8-4c9e-aff5-78d3dbbc0199" (UID: "8de060ca-04d8-4c9e-aff5-78d3dbbc0199"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:16.066424 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.066405    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-envoy" pod "8de060ca-04d8-4c9e-aff5-78d3dbbc0199" (UID: "8de060ca-04d8-4c9e-aff5-78d3dbbc0199"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:16.066723 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.066704    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-podinfo" pod "8de060ca-04d8-4c9e-aff5-78d3dbbc0199" (UID: "8de060ca-04d8-4c9e-aff5-78d3dbbc0199"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:54:16.066723 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.066709    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-kube-api-access-cdd5p" pod "8de060ca-04d8-4c9e-aff5-78d3dbbc0199" (UID: "8de060ca-04d8-4c9e-aff5-78d3dbbc0199"). InnerVolumeSpecName "kube-api-access-cdd5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:16.166535 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.166500    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:16.166535 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.166528    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:16.166535 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.166539    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:16.166758 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.166550    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdd5p\" (UniqueName: \"kubernetes.io/projected/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-kube-api-access-cdd5p\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:16.166758 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.166559    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de060ca-04d8-4c9e-aff5-78d3dbbc0199-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:16.550578 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.550543    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"]
Oct 30 19:54:16.550799 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:16.550752    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp" podUID="5e94be7d-8ae8-4249-9270-691667f8b406" containerName="maas-api" containerID="cri-o://3e85e704c4051d54abad217cdc99b9b77c62a88e655001b254476d3a8d44938b" gracePeriod=30
Oct 30 19:54:17.031500 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.031467    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj" event={"ID":"8de060ca-04d8-4c9e-aff5-78d3dbbc0199","Type":"ContainerDied","Data":"40fdce100fb16bee61d8e4a8156d37958e4a05d0029e9a726768cba25b18d56c"}
Oct 30 19:54:17.032739 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.031480    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"
Oct 30 19:54:17.033334 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.033311    6030 scope.go:122] "RemoveContainer" containerID="3f940f3d46e522c8876bedb87919cefab88c9e573f3af29248eeb2ebbdbd595c"
Oct 30 19:54:17.033758 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.033732    6030 generic.go:356] "Generic (PLEG): container finished" podID="5e94be7d-8ae8-4249-9270-691667f8b406" containerID="3e85e704c4051d54abad217cdc99b9b77c62a88e655001b254476d3a8d44938b" exitCode=0
Oct 30 19:54:17.034022 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.033999    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp" event={"ID":"5e94be7d-8ae8-4249-9270-691667f8b406","Type":"ContainerDied","Data":"3e85e704c4051d54abad217cdc99b9b77c62a88e655001b254476d3a8d44938b"}
Oct 30 19:54:17.063849 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.063827    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:54:17.082907 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.082872    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"]
Oct 30 19:54:17.088904 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.088881    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-59e21a-openshift-default-5ddffb659-j8rjj"]
Oct 30 19:54:17.201314 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.201281    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8de060ca-04d8-4c9e-aff5-78d3dbbc0199" path="/var/lib/kubelet/pods/8de060ca-04d8-4c9e-aff5-78d3dbbc0199/volumes"
Oct 30 19:54:17.231394 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.231360    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e94be7d-8ae8-4249-9270-691667f8b406-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e94be7d-8ae8-4249-9270-691667f8b406-metrics-certs\") pod \"5e94be7d-8ae8-4249-9270-691667f8b406\" (UID: \"5e94be7d-8ae8-4249-9270-691667f8b406\") "
Oct 30 19:54:17.231667 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.231638    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e94be7d-8ae8-4249-9270-691667f8b406-kube-api-access-nwdpb\" (UniqueName: \"kubernetes.io/projected/5e94be7d-8ae8-4249-9270-691667f8b406-kube-api-access-nwdpb\") pod \"5e94be7d-8ae8-4249-9270-691667f8b406\" (UID: \"5e94be7d-8ae8-4249-9270-691667f8b406\") "
Oct 30 19:54:17.231811 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.231704    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e94be7d-8ae8-4249-9270-691667f8b406-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e94be7d-8ae8-4249-9270-691667f8b406-maas-api-tls\") pod \"5e94be7d-8ae8-4249-9270-691667f8b406\" (UID: \"5e94be7d-8ae8-4249-9270-691667f8b406\") "
Oct 30 19:54:17.231876 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.231845    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e94be7d-8ae8-4249-9270-691667f8b406-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e94be7d-8ae8-4249-9270-691667f8b406-openshift-service-ca\") pod \"5e94be7d-8ae8-4249-9270-691667f8b406\" (UID: \"5e94be7d-8ae8-4249-9270-691667f8b406\") "
Oct 30 19:54:17.232850 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.232633    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e94be7d-8ae8-4249-9270-691667f8b406-openshift-service-ca" pod "5e94be7d-8ae8-4249-9270-691667f8b406" (UID: "5e94be7d-8ae8-4249-9270-691667f8b406"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:17.235118 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.235093    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e94be7d-8ae8-4249-9270-691667f8b406-kube-api-access-nwdpb" pod "5e94be7d-8ae8-4249-9270-691667f8b406" (UID: "5e94be7d-8ae8-4249-9270-691667f8b406"). InnerVolumeSpecName "kube-api-access-nwdpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:17.236162 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.236136    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e94be7d-8ae8-4249-9270-691667f8b406-maas-api-tls" pod "5e94be7d-8ae8-4249-9270-691667f8b406" (UID: "5e94be7d-8ae8-4249-9270-691667f8b406"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:17.239164 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.239143    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e94be7d-8ae8-4249-9270-691667f8b406-metrics-certs" pod "5e94be7d-8ae8-4249-9270-691667f8b406" (UID: "5e94be7d-8ae8-4249-9270-691667f8b406"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:17.333453 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.333369    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e94be7d-8ae8-4249-9270-691667f8b406-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:17.333453 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.333398    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e94be7d-8ae8-4249-9270-691667f8b406-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:17.333453 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.333410    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwdpb\" (UniqueName: \"kubernetes.io/projected/5e94be7d-8ae8-4249-9270-691667f8b406-kube-api-access-nwdpb\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:17.333453 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:17.333426    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e94be7d-8ae8-4249-9270-691667f8b406-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:18.009245 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009211    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"]
Oct 30 19:54:18.009726 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009704    6030 cpu_manager.go:414] "containerMap: removing container" podUID="8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5" containerName="payload-processing"
Oct 30 19:54:18.009726 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009718    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5" containerName="payload-processing"
Oct 30 19:54:18.009816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009737    6030 cpu_manager.go:414] "containerMap: removing container" podUID="8de060ca-04d8-4c9e-aff5-78d3dbbc0199" containerName="istio-proxy"
Oct 30 19:54:18.009816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009742    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de060ca-04d8-4c9e-aff5-78d3dbbc0199" containerName="istio-proxy"
Oct 30 19:54:18.009816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009752    6030 cpu_manager.go:414] "containerMap: removing container" podUID="8de060ca-04d8-4c9e-aff5-78d3dbbc0199" containerName="istio-proxy"
Oct 30 19:54:18.009816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009758    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de060ca-04d8-4c9e-aff5-78d3dbbc0199" containerName="istio-proxy"
Oct 30 19:54:18.009816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009764    6030 cpu_manager.go:414] "containerMap: removing container" podUID="5e94be7d-8ae8-4249-9270-691667f8b406" containerName="maas-api"
Oct 30 19:54:18.009816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009769    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e94be7d-8ae8-4249-9270-691667f8b406" containerName="maas-api"
Oct 30 19:54:18.009816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009786    6030 cpu_manager.go:414] "containerMap: removing container" podUID="842afcb7-9473-4e5a-b9fa-3486a236c1a5" containerName="istio-proxy"
Oct 30 19:54:18.009816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009791    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842afcb7-9473-4e5a-b9fa-3486a236c1a5" containerName="istio-proxy"
Oct 30 19:54:18.009816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009804    6030 cpu_manager.go:414] "containerMap: removing container" podUID="8efb9f1a-3f05-4ddd-9ab5-7086f717524c" containerName="payload-pre-processing"
Oct 30 19:54:18.009816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009809    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efb9f1a-3f05-4ddd-9ab5-7086f717524c" containerName="payload-pre-processing"
Oct 30 19:54:18.009816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009816    6030 cpu_manager.go:414] "containerMap: removing container" podUID="cac65540-89ac-4591-b288-14f8ac45be99" containerName="maas-api"
Oct 30 19:54:18.009816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009821    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac65540-89ac-4591-b288-14f8ac45be99" containerName="maas-api"
Oct 30 19:54:18.010205 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009832    6030 cpu_manager.go:414] "containerMap: removing container" podUID="14294872-1aaf-4b7d-b2ab-f80fbf7db165" containerName="payload-pre-processing"
Oct 30 19:54:18.010205 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009837    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14294872-1aaf-4b7d-b2ab-f80fbf7db165" containerName="payload-pre-processing"
Oct 30 19:54:18.010205 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009906    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de060ca-04d8-4c9e-aff5-78d3dbbc0199" containerName="istio-proxy"
Oct 30 19:54:18.010205 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009913    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efb9f1a-3f05-4ddd-9ab5-7086f717524c" containerName="payload-pre-processing"
Oct 30 19:54:18.010205 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009920    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac65540-89ac-4591-b288-14f8ac45be99" containerName="maas-api"
Oct 30 19:54:18.010205 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009927    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de060ca-04d8-4c9e-aff5-78d3dbbc0199" containerName="istio-proxy"
Oct 30 19:54:18.010205 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009934    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="14294872-1aaf-4b7d-b2ab-f80fbf7db165" containerName="payload-pre-processing"
Oct 30 19:54:18.010205 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009939    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e94be7d-8ae8-4249-9270-691667f8b406" containerName="maas-api"
Oct 30 19:54:18.010205 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009948    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8eecbe-1cc4-425f-b0b7-cfcb00434bd5" containerName="payload-processing"
Oct 30 19:54:18.010205 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.009954    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="842afcb7-9473-4e5a-b9fa-3486a236c1a5" containerName="istio-proxy"
Oct 30 19:54:18.013325 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.013305    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.029411 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.029380    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"]
Oct 30 19:54:18.043834 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.043749    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.043834 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.043819    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.044373 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.043891    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.044373 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.043921    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp" event={"ID":"5e94be7d-8ae8-4249-9270-691667f8b406","Type":"ContainerDied","Data":"73e422ba56befd939066755575520ac7d08c2d9963b182ef7890a1e1175f2b08"}
Oct 30 19:54:18.044373 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.043950    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.044373 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.044002    6030 scope.go:122] "RemoveContainer" containerID="3e85e704c4051d54abad217cdc99b9b77c62a88e655001b254476d3a8d44938b"
Oct 30 19:54:18.044373 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.044035    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.044373 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.044167    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.044373 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.044178    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"
Oct 30 19:54:18.044373 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.044201    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.044373 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.044230    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7vms\" (UniqueName: \"kubernetes.io/projected/a7b54a5e-b3af-466f-990e-21e6b60ae71b-kube-api-access-b7vms\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.044373 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.044278    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.082001 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.081974    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"]
Oct 30 19:54:18.085438 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.085412    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ee4af9-667c85f5b4-xbgjp"]
Oct 30 19:54:18.145803 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.145769    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.146001 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.145848    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.146001 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.145880    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.146001 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.145899    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7vms\" (UniqueName: \"kubernetes.io/projected/a7b54a5e-b3af-466f-990e-21e6b60ae71b-kube-api-access-b7vms\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.146001 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.145928    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.147416 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.145994    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.147416 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.146090    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.147416 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.146222    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.147416 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.146279    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.147416 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.146326    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.147416 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.146588    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.147416 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.146664    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.147416 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.147097    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.147416 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.147157    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.148577 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.148556    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.149113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.149094    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.164583 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.164558    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.164696 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.164603    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7vms\" (UniqueName: \"kubernetes.io/projected/a7b54a5e-b3af-466f-990e-21e6b60ae71b-kube-api-access-b7vms\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.332323 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.332235    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:18.642403 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.642378    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"]
Oct 30 19:54:18.644107 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:54:18.644082    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b54a5e_b3af_466f_990e_21e6b60ae71b.slice/crio-630d82043e5465ef6163b3328c9dbccf689b53f943ebe79336bfe519242017e3 WatchSource:0}: Error finding container 630d82043e5465ef6163b3328c9dbccf689b53f943ebe79336bfe519242017e3: Status 404 returned error can't find the container with id 630d82043e5465ef6163b3328c9dbccf689b53f943ebe79336bfe519242017e3
Oct 30 19:54:18.645990 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.645959    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:54:18.646073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.646058    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:54:18.646119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.646093    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:54:18.650455 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.650432    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"]
Oct 30 19:54:18.655462 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.655446    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"
Oct 30 19:54:18.664639 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.664584    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d2dd78\""
Oct 30 19:54:18.664780 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.664712    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d2dd78-dockercfg-rbwbg\""
Oct 30 19:54:18.665736 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.665715    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"]
Oct 30 19:54:18.690349 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.690320    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"]
Oct 30 19:54:18.697533 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.697507    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"
Oct 30 19:54:18.704637 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.704611    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"]
Oct 30 19:54:18.773083 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.757795    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e770bf-b4c1-47c3-b850-92c9d42dd9ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h\" (UID: \"e9e770bf-b4c1-47c3-b850-92c9d42dd9ba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"
Oct 30 19:54:18.773083 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.757902    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m852k\" (UniqueName: \"kubernetes.io/projected/d61e89eb-9e7b-413d-9558-2382b36f29d9-kube-api-access-m852k\") pod \"payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp\" (UID: \"d61e89eb-9e7b-413d-9558-2382b36f29d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"
Oct 30 19:54:18.773083 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.758070    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9lxs\" (UniqueName: \"kubernetes.io/projected/e9e770bf-b4c1-47c3-b850-92c9d42dd9ba-kube-api-access-r9lxs\") pod \"payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h\" (UID: \"e9e770bf-b4c1-47c3-b850-92c9d42dd9ba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"
Oct 30 19:54:18.773083 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.758114    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61e89eb-9e7b-413d-9558-2382b36f29d9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp\" (UID: \"d61e89eb-9e7b-413d-9558-2382b36f29d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"
Oct 30 19:54:18.911582 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.901243    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9lxs\" (UniqueName: \"kubernetes.io/projected/e9e770bf-b4c1-47c3-b850-92c9d42dd9ba-kube-api-access-r9lxs\") pod \"payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h\" (UID: \"e9e770bf-b4c1-47c3-b850-92c9d42dd9ba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"
Oct 30 19:54:18.911582 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.901331    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61e89eb-9e7b-413d-9558-2382b36f29d9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp\" (UID: \"d61e89eb-9e7b-413d-9558-2382b36f29d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"
Oct 30 19:54:18.911582 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.901455    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e770bf-b4c1-47c3-b850-92c9d42dd9ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h\" (UID: \"e9e770bf-b4c1-47c3-b850-92c9d42dd9ba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"
Oct 30 19:54:18.911582 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.901538    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m852k\" (UniqueName: \"kubernetes.io/projected/d61e89eb-9e7b-413d-9558-2382b36f29d9-kube-api-access-m852k\") pod \"payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp\" (UID: \"d61e89eb-9e7b-413d-9558-2382b36f29d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"
Oct 30 19:54:18.911582 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.902828    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61e89eb-9e7b-413d-9558-2382b36f29d9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp\" (UID: \"d61e89eb-9e7b-413d-9558-2382b36f29d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"
Oct 30 19:54:18.911582 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.903626    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e770bf-b4c1-47c3-b850-92c9d42dd9ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h\" (UID: \"e9e770bf-b4c1-47c3-b850-92c9d42dd9ba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"
Oct 30 19:54:18.950385 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.950102    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m852k\" (UniqueName: \"kubernetes.io/projected/d61e89eb-9e7b-413d-9558-2382b36f29d9-kube-api-access-m852k\") pod \"payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp\" (UID: \"d61e89eb-9e7b-413d-9558-2382b36f29d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"
Oct 30 19:54:18.950533 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.950452    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9lxs\" (UniqueName: \"kubernetes.io/projected/e9e770bf-b4c1-47c3-b850-92c9d42dd9ba-kube-api-access-r9lxs\") pod \"payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h\" (UID: \"e9e770bf-b4c1-47c3-b850-92c9d42dd9ba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"
Oct 30 19:54:18.972456 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:18.972424    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"
Oct 30 19:54:19.022534 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:19.022313    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"
Oct 30 19:54:19.070692 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:19.069617    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2" event={"ID":"a7b54a5e-b3af-466f-990e-21e6b60ae71b","Type":"ContainerStarted","Data":"859362de05e709033c670d35920b9610b6533473cefb1fe7d685988a2345d19e"}
Oct 30 19:54:19.070692 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:19.069682    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2" event={"ID":"a7b54a5e-b3af-466f-990e-21e6b60ae71b","Type":"ContainerStarted","Data":"630d82043e5465ef6163b3328c9dbccf689b53f943ebe79336bfe519242017e3"}
Oct 30 19:54:19.102245 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:19.102192    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2" podStartSLOduration=2.102171783 podStartE2EDuration="2.102171783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:19.102007143 +0000 UTC m=+1338.017614003" watchObservedRunningTime="2026-10-30 19:54:19.102171783 +0000 UTC m=+1338.017778643"
Oct 30 19:54:19.209731 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:19.209373    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e94be7d-8ae8-4249-9270-691667f8b406" path="/var/lib/kubelet/pods/5e94be7d-8ae8-4249-9270-691667f8b406/volumes"
Oct 30 19:54:19.265596 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:54:19.265548    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61e89eb_9e7b_413d_9558_2382b36f29d9.slice/crio-8d103582bcad7e423b842e7a28541c146e0ada9afb4ca4b06b1bf19567e7f59f WatchSource:0}: Error finding container 8d103582bcad7e423b842e7a28541c146e0ada9afb4ca4b06b1bf19567e7f59f: Status 404 returned error can't find the container with id 8d103582bcad7e423b842e7a28541c146e0ada9afb4ca4b06b1bf19567e7f59f
Oct 30 19:54:19.269850 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:19.269212    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"]
Oct 30 19:54:19.330687 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:19.330663    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"]
Oct 30 19:54:19.332093 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:54:19.332064    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e770bf_b4c1_47c3_b850_92c9d42dd9ba.slice/crio-67f182bd03cded515b5134e96e9a3cb9d184050e19b72522f538f38cf1e880e6 WatchSource:0}: Error finding container 67f182bd03cded515b5134e96e9a3cb9d184050e19b72522f538f38cf1e880e6: Status 404 returned error can't find the container with id 67f182bd03cded515b5134e96e9a3cb9d184050e19b72522f538f38cf1e880e6
Oct 30 19:54:19.332629 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:19.332605    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:19.334181 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:19.334158    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:54:19.334275 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:19.334201    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2" podUID="a7b54a5e-b3af-466f-990e-21e6b60ae71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:54:20.102881 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:20.102817    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h" event={"ID":"e9e770bf-b4c1-47c3-b850-92c9d42dd9ba","Type":"ContainerStarted","Data":"52cf17c020737c0f119752453489c10d5a541be633b29e1c4b85097935efd570"}
Oct 30 19:54:20.103414 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:20.102950    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h" event={"ID":"e9e770bf-b4c1-47c3-b850-92c9d42dd9ba","Type":"ContainerStarted","Data":"67f182bd03cded515b5134e96e9a3cb9d184050e19b72522f538f38cf1e880e6"}
Oct 30 19:54:20.103414 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:20.102980    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"
Oct 30 19:54:20.104928 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:20.104899    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp" event={"ID":"d61e89eb-9e7b-413d-9558-2382b36f29d9","Type":"ContainerStarted","Data":"2015f29ca555e14b476df269cbd8b3f1a0ab3a2d9ba490db66a5fb5e30282750"}
Oct 30 19:54:20.105086 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:20.104952    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp" event={"ID":"d61e89eb-9e7b-413d-9558-2382b36f29d9","Type":"ContainerStarted","Data":"8d103582bcad7e423b842e7a28541c146e0ada9afb4ca4b06b1bf19567e7f59f"}
Oct 30 19:54:20.105166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:20.105154    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"
Oct 30 19:54:20.127279 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:20.127227    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h" podStartSLOduration=2.127213251 podStartE2EDuration="2.127213251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:20.126597353 +0000 UTC m=+1339.042204210" watchObservedRunningTime="2026-10-30 19:54:20.127213251 +0000 UTC m=+1339.042820109"
Oct 30 19:54:20.148121 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:20.148071    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp" podStartSLOduration=2.148034927 podStartE2EDuration="2.148034927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:20.146320359 +0000 UTC m=+1339.061927215" watchObservedRunningTime="2026-10-30 19:54:20.148034927 +0000 UTC m=+1339.063641783"
Oct 30 19:54:20.333583 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:20.333545    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:54:20.334661 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:20.334628    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2" podUID="a7b54a5e-b3af-466f-990e-21e6b60ae71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:54:21.339252 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:21.339221    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:54:21.339858 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:21.339827    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2" podUID="a7b54a5e-b3af-466f-990e-21e6b60ae71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:54:22.332791 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:22.332754    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:54:22.333146 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:22.333112    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2" podUID="a7b54a5e-b3af-466f-990e-21e6b60ae71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:54:23.127269 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.127240    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"]
Oct 30 19:54:23.131389 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.131370    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"
Oct 30 19:54:23.139691 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.139662    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-55a8e6\""
Oct 30 19:54:23.139803 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.139783    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-55a8e6-dockercfg-gwt87\""
Oct 30 19:54:23.143663 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.143638    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"]
Oct 30 19:54:23.203229 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.203179    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zpwt\" (UniqueName: \"kubernetes.io/projected/273ea5ae-d8b5-4f08-b2e2-f7deb26def92-kube-api-access-9zpwt\") pod \"payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx\" (UID: \"273ea5ae-d8b5-4f08-b2e2-f7deb26def92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"
Oct 30 19:54:23.203436 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.203261    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273ea5ae-d8b5-4f08-b2e2-f7deb26def92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx\" (UID: \"273ea5ae-d8b5-4f08-b2e2-f7deb26def92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"
Oct 30 19:54:23.311071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.298750    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"]
Oct 30 19:54:23.327069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.317829    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zpwt\" (UniqueName: \"kubernetes.io/projected/273ea5ae-d8b5-4f08-b2e2-f7deb26def92-kube-api-access-9zpwt\") pod \"payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx\" (UID: \"273ea5ae-d8b5-4f08-b2e2-f7deb26def92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"
Oct 30 19:54:23.327069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.317905    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273ea5ae-d8b5-4f08-b2e2-f7deb26def92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx\" (UID: \"273ea5ae-d8b5-4f08-b2e2-f7deb26def92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"
Oct 30 19:54:23.335069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.331525    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"
Oct 30 19:54:23.344065 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.341071    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273ea5ae-d8b5-4f08-b2e2-f7deb26def92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx\" (UID: \"273ea5ae-d8b5-4f08-b2e2-f7deb26def92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"
Oct 30 19:54:23.353114 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.351702    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:54:23.353114 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.351772    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2" podUID="a7b54a5e-b3af-466f-990e-21e6b60ae71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:54:23.367069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.360486    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"]
Oct 30 19:54:23.409119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.408882    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zpwt\" (UniqueName: \"kubernetes.io/projected/273ea5ae-d8b5-4f08-b2e2-f7deb26def92-kube-api-access-9zpwt\") pod \"payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx\" (UID: \"273ea5ae-d8b5-4f08-b2e2-f7deb26def92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"
Oct 30 19:54:23.436572 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.433427    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9188e0-7232-4f19-8d67-82758f1db85c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w\" (UID: \"0e9188e0-7232-4f19-8d67-82758f1db85c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"
Oct 30 19:54:23.436572 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.433582    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btnmc\" (UniqueName: \"kubernetes.io/projected/0e9188e0-7232-4f19-8d67-82758f1db85c-kube-api-access-btnmc\") pod \"payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w\" (UID: \"0e9188e0-7232-4f19-8d67-82758f1db85c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"
Oct 30 19:54:23.452395 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.451136    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"
Oct 30 19:54:23.535112 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.535030    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9188e0-7232-4f19-8d67-82758f1db85c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w\" (UID: \"0e9188e0-7232-4f19-8d67-82758f1db85c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"
Oct 30 19:54:23.535259 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.535248    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btnmc\" (UniqueName: \"kubernetes.io/projected/0e9188e0-7232-4f19-8d67-82758f1db85c-kube-api-access-btnmc\") pod \"payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w\" (UID: \"0e9188e0-7232-4f19-8d67-82758f1db85c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"
Oct 30 19:54:23.536585 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.536561    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9188e0-7232-4f19-8d67-82758f1db85c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w\" (UID: \"0e9188e0-7232-4f19-8d67-82758f1db85c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"
Oct 30 19:54:23.552134 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.552106    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btnmc\" (UniqueName: \"kubernetes.io/projected/0e9188e0-7232-4f19-8d67-82758f1db85c-kube-api-access-btnmc\") pod \"payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w\" (UID: \"0e9188e0-7232-4f19-8d67-82758f1db85c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"
Oct 30 19:54:23.826269 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.826237    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"]
Oct 30 19:54:23.827533 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:23.827313    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"
Oct 30 19:54:23.828125 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:54:23.828089    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273ea5ae_d8b5_4f08_b2e2_f7deb26def92.slice/crio-e9694c67ee96d505d281335536125bf458e61bc5f7b148ee305e46bed15952de WatchSource:0}: Error finding container e9694c67ee96d505d281335536125bf458e61bc5f7b148ee305e46bed15952de: Status 404 returned error can't find the container with id e9694c67ee96d505d281335536125bf458e61bc5f7b148ee305e46bed15952de
Oct 30 19:54:24.136880 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:24.136845    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx" event={"ID":"273ea5ae-d8b5-4f08-b2e2-f7deb26def92","Type":"ContainerStarted","Data":"4e2a89fa64e0611e81f709fd00eba87f4729b3ef9d29701981c3ee0459633bc0"}
Oct 30 19:54:24.137329 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:24.136897    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx" event={"ID":"273ea5ae-d8b5-4f08-b2e2-f7deb26def92","Type":"ContainerStarted","Data":"e9694c67ee96d505d281335536125bf458e61bc5f7b148ee305e46bed15952de"}
Oct 30 19:54:24.137434 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:24.137394    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"
Oct 30 19:54:24.168445 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:24.168396    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx" podStartSLOduration=1.168378642 podStartE2EDuration="1.168378642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:24.167878044 +0000 UTC m=+1343.083484899" watchObservedRunningTime="2026-10-30 19:54:24.168378642 +0000 UTC m=+1343.083985497"
Oct 30 19:54:24.227966 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:24.227932    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"]
Oct 30 19:54:24.231239 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:54:24.231210    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9188e0_7232_4f19_8d67_82758f1db85c.slice/crio-d012399913b9ce806f1de797479ca6b1fbb4fe4fb7fd2d8b48c9b4854bcbbb91 WatchSource:0}: Error finding container d012399913b9ce806f1de797479ca6b1fbb4fe4fb7fd2d8b48c9b4854bcbbb91: Status 404 returned error can't find the container with id d012399913b9ce806f1de797479ca6b1fbb4fe4fb7fd2d8b48c9b4854bcbbb91
Oct 30 19:54:24.347068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:24.341348    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:54:24.347068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:24.341406    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2" podUID="a7b54a5e-b3af-466f-990e-21e6b60ae71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:54:25.144054 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:25.144009    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w" event={"ID":"0e9188e0-7232-4f19-8d67-82758f1db85c","Type":"ContainerStarted","Data":"220177dd98bd2800306d0febcd573c94c06d4a51dfa31041fb77a46dce916d1d"}
Oct 30 19:54:25.144537 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:25.144077    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w" event={"ID":"0e9188e0-7232-4f19-8d67-82758f1db85c","Type":"ContainerStarted","Data":"d012399913b9ce806f1de797479ca6b1fbb4fe4fb7fd2d8b48c9b4854bcbbb91"}
Oct 30 19:54:25.144537 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:25.144172    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"
Oct 30 19:54:25.167998 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:25.167948    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w" podStartSLOduration=2.167934 podStartE2EDuration="2.167934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:25.166109929 +0000 UTC m=+1344.081716781" watchObservedRunningTime="2026-10-30 19:54:25.167934 +0000 UTC m=+1344.083540918"
Oct 30 19:54:25.341072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:25.336163    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:54:25.341072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:25.336222    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2" podUID="a7b54a5e-b3af-466f-990e-21e6b60ae71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:54:26.849844 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:26.849800    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:26.850473 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:26.850260    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:26.851440 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:26.851419    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 19:54:30.691068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:30.690587    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"]
Oct 30 19:54:30.723067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:30.722292    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"]
Oct 30 19:54:30.723248 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:30.723169    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:54:30.731656 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:30.731632    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d2dd78-serving-cert\""
Oct 30 19:54:30.834102 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:30.833672    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94e0001-0f96-4e02-bbba-fd81d82584e9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk\" (UID: \"d94e0001-0f96-4e02-bbba-fd81d82584e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:54:30.834516 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:30.834466    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d94e0001-0f96-4e02-bbba-fd81d82584e9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk\" (UID: \"d94e0001-0f96-4e02-bbba-fd81d82584e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:54:30.834630 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:30.834611    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kplj7\" (UniqueName: \"kubernetes.io/projected/d94e0001-0f96-4e02-bbba-fd81d82584e9-kube-api-access-kplj7\") pod \"maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk\" (UID: \"d94e0001-0f96-4e02-bbba-fd81d82584e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:54:30.834679 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:30.834653    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d94e0001-0f96-4e02-bbba-fd81d82584e9-metrics-certs\") pod \"maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk\" (UID: \"d94e0001-0f96-4e02-bbba-fd81d82584e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:54:30.935702 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:30.935670    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94e0001-0f96-4e02-bbba-fd81d82584e9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk\" (UID: \"d94e0001-0f96-4e02-bbba-fd81d82584e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:54:30.935912 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:30.935733    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d94e0001-0f96-4e02-bbba-fd81d82584e9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk\" (UID: \"d94e0001-0f96-4e02-bbba-fd81d82584e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:54:30.935912 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:30.935786    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kplj7\" (UniqueName: \"kubernetes.io/projected/d94e0001-0f96-4e02-bbba-fd81d82584e9-kube-api-access-kplj7\") pod \"maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk\" (UID: \"d94e0001-0f96-4e02-bbba-fd81d82584e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:54:30.935912 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:30.935808    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d94e0001-0f96-4e02-bbba-fd81d82584e9-metrics-certs\") pod \"maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk\" (UID: \"d94e0001-0f96-4e02-bbba-fd81d82584e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:54:30.936129 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:54:30.935907    6030 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-serving-cert: secret "maas-api-e2e-praxis-a-d2dd78-serving-cert" not found
Oct 30 19:54:30.936129 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:54:30.935991    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d94e0001-0f96-4e02-bbba-fd81d82584e9-maas-api-tls podName:d94e0001-0f96-4e02-bbba-fd81d82584e9 nodeName:}" failed. No retries permitted until 2026-10-30 19:54:31.435967829 +0000 UTC m=+1350.351574681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d94e0001-0f96-4e02-bbba-fd81d82584e9-maas-api-tls") pod "maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk" (UID: "d94e0001-0f96-4e02-bbba-fd81d82584e9") : secret "maas-api-e2e-praxis-a-d2dd78-serving-cert" not found
Oct 30 19:54:30.936448 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:30.936427    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94e0001-0f96-4e02-bbba-fd81d82584e9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk\" (UID: \"d94e0001-0f96-4e02-bbba-fd81d82584e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:54:30.938531 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:30.938508    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d94e0001-0f96-4e02-bbba-fd81d82584e9-metrics-certs\") pod \"maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk\" (UID: \"d94e0001-0f96-4e02-bbba-fd81d82584e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:54:30.949790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:30.949732    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kplj7\" (UniqueName: \"kubernetes.io/projected/d94e0001-0f96-4e02-bbba-fd81d82584e9-kube-api-access-kplj7\") pod \"maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk\" (UID: \"d94e0001-0f96-4e02-bbba-fd81d82584e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:54:31.115824 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:31.115792    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"
Oct 30 19:54:31.116404 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:31.116376    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"
Oct 30 19:54:31.444069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:31.443928    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d94e0001-0f96-4e02-bbba-fd81d82584e9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk\" (UID: \"d94e0001-0f96-4e02-bbba-fd81d82584e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:54:31.462132 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:31.460653    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d94e0001-0f96-4e02-bbba-fd81d82584e9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk\" (UID: \"d94e0001-0f96-4e02-bbba-fd81d82584e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:54:31.645389 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:31.645344    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:54:31.963181 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:31.963150    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"]
Oct 30 19:54:31.964928 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:54:31.964903    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94e0001_0f96_4e02_bbba_fd81d82584e9.slice/crio-dff5e2f3e3a5c3100d7d7f641351b1c14503a7d2a54dd8a80e1bc8859eb4fbdf WatchSource:0}: Error finding container dff5e2f3e3a5c3100d7d7f641351b1c14503a7d2a54dd8a80e1bc8859eb4fbdf: Status 404 returned error can't find the container with id dff5e2f3e3a5c3100d7d7f641351b1c14503a7d2a54dd8a80e1bc8859eb4fbdf
Oct 30 19:54:32.272067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:32.271954    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk" event={"ID":"d94e0001-0f96-4e02-bbba-fd81d82584e9","Type":"ContainerStarted","Data":"dff5e2f3e3a5c3100d7d7f641351b1c14503a7d2a54dd8a80e1bc8859eb4fbdf"}
Oct 30 19:54:33.279454 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:33.279423    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk" event={"ID":"d94e0001-0f96-4e02-bbba-fd81d82584e9","Type":"ContainerStarted","Data":"0699336a3b105d30880d4679a12a7e9595f2d4c8e6524cdd0c38680c2e67fdd7"}
Oct 30 19:54:33.279880 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:33.279540    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:54:33.309681 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:33.309624    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk" podStartSLOduration=2.851113839 podStartE2EDuration="3.30960997s" totalImagesPullingTime="458.496131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:33.307395311 +0000 UTC m=+1352.223002168" watchObservedRunningTime="2026-10-30 19:54:33.30960997 +0000 UTC m=+1352.225216826"
Oct 30 19:54:34.306207 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:54:34.306164    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:54:35.163321 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:35.163241    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"
Oct 30 19:54:36.152096 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:36.152029    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"
Oct 30 19:54:39.294914 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:39.294886    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:54:55.750497 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:55.750455    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"]
Oct 30 19:54:55.751253 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:55.750756    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2" podUID="b59ec827-e83b-4e1f-b138-bccae9cc5356" containerName="maas-api" containerID="cri-o://412b579e110c38596854bdf12e94e9c35b6ef4fb5cacb8c31ea09e36f9eb5983" gracePeriod=30
Oct 30 19:54:56.122522 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.122481    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:54:56.169078 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.169012    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b59ec827-e83b-4e1f-b138-bccae9cc5356-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59ec827-e83b-4e1f-b138-bccae9cc5356-openshift-service-ca\") pod \"b59ec827-e83b-4e1f-b138-bccae9cc5356\" (UID: \"b59ec827-e83b-4e1f-b138-bccae9cc5356\") "
Oct 30 19:54:56.169269 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.169109    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b59ec827-e83b-4e1f-b138-bccae9cc5356-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59ec827-e83b-4e1f-b138-bccae9cc5356-metrics-certs\") pod \"b59ec827-e83b-4e1f-b138-bccae9cc5356\" (UID: \"b59ec827-e83b-4e1f-b138-bccae9cc5356\") "
Oct 30 19:54:56.169269 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.169179    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b59ec827-e83b-4e1f-b138-bccae9cc5356-kube-api-access-lxw2c\" (UniqueName: \"kubernetes.io/projected/b59ec827-e83b-4e1f-b138-bccae9cc5356-kube-api-access-lxw2c\") pod \"b59ec827-e83b-4e1f-b138-bccae9cc5356\" (UID: \"b59ec827-e83b-4e1f-b138-bccae9cc5356\") "
Oct 30 19:54:56.169396 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.169291    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b59ec827-e83b-4e1f-b138-bccae9cc5356-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59ec827-e83b-4e1f-b138-bccae9cc5356-maas-api-tls\") pod \"b59ec827-e83b-4e1f-b138-bccae9cc5356\" (UID: \"b59ec827-e83b-4e1f-b138-bccae9cc5356\") "
Oct 30 19:54:56.169707 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.169667    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b59ec827-e83b-4e1f-b138-bccae9cc5356-openshift-service-ca" pod "b59ec827-e83b-4e1f-b138-bccae9cc5356" (UID: "b59ec827-e83b-4e1f-b138-bccae9cc5356"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:56.169948 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.169925    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59ec827-e83b-4e1f-b138-bccae9cc5356-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:56.173873 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.173814    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b59ec827-e83b-4e1f-b138-bccae9cc5356-maas-api-tls" pod "b59ec827-e83b-4e1f-b138-bccae9cc5356" (UID: "b59ec827-e83b-4e1f-b138-bccae9cc5356"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:56.174237 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.174210    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59ec827-e83b-4e1f-b138-bccae9cc5356-kube-api-access-lxw2c" pod "b59ec827-e83b-4e1f-b138-bccae9cc5356" (UID: "b59ec827-e83b-4e1f-b138-bccae9cc5356"). InnerVolumeSpecName "kube-api-access-lxw2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:56.177835 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.177807    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b59ec827-e83b-4e1f-b138-bccae9cc5356-metrics-certs" pod "b59ec827-e83b-4e1f-b138-bccae9cc5356" (UID: "b59ec827-e83b-4e1f-b138-bccae9cc5356"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:56.287089 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.284184    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59ec827-e83b-4e1f-b138-bccae9cc5356-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:56.287089 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.284227    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxw2c\" (UniqueName: \"kubernetes.io/projected/b59ec827-e83b-4e1f-b138-bccae9cc5356-kube-api-access-lxw2c\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:56.287089 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.284244    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59ec827-e83b-4e1f-b138-bccae9cc5356-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:56.521493 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.521462    6030 generic.go:356] "Generic (PLEG): container finished" podID="b59ec827-e83b-4e1f-b138-bccae9cc5356" containerID="412b579e110c38596854bdf12e94e9c35b6ef4fb5cacb8c31ea09e36f9eb5983" exitCode=0
Oct 30 19:54:56.521674 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.521511    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"
Oct 30 19:54:56.521674 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.521551    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2" event={"ID":"b59ec827-e83b-4e1f-b138-bccae9cc5356","Type":"ContainerDied","Data":"412b579e110c38596854bdf12e94e9c35b6ef4fb5cacb8c31ea09e36f9eb5983"}
Oct 30 19:54:56.521674 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.521607    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2" event={"ID":"b59ec827-e83b-4e1f-b138-bccae9cc5356","Type":"ContainerDied","Data":"fe9d405c2a9e0a41342d371f7560b96de1ff6a07530eb9b443b4373088a5c621"}
Oct 30 19:54:56.521674 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.521632    6030 scope.go:122] "RemoveContainer" containerID="412b579e110c38596854bdf12e94e9c35b6ef4fb5cacb8c31ea09e36f9eb5983"
Oct 30 19:54:56.532177 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.532034    6030 scope.go:122] "RemoveContainer" containerID="412b579e110c38596854bdf12e94e9c35b6ef4fb5cacb8c31ea09e36f9eb5983"
Oct 30 19:54:56.532371 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:54:56.532348    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412b579e110c38596854bdf12e94e9c35b6ef4fb5cacb8c31ea09e36f9eb5983\": container with ID starting with 412b579e110c38596854bdf12e94e9c35b6ef4fb5cacb8c31ea09e36f9eb5983 not found: ID does not exist" containerID="412b579e110c38596854bdf12e94e9c35b6ef4fb5cacb8c31ea09e36f9eb5983"
Oct 30 19:54:56.532428 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.532381    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412b579e110c38596854bdf12e94e9c35b6ef4fb5cacb8c31ea09e36f9eb5983"} err="failed to get container status \"412b579e110c38596854bdf12e94e9c35b6ef4fb5cacb8c31ea09e36f9eb5983\": rpc error: code = NotFound desc = could not find container \"412b579e110c38596854bdf12e94e9c35b6ef4fb5cacb8c31ea09e36f9eb5983\": container with ID starting with 412b579e110c38596854bdf12e94e9c35b6ef4fb5cacb8c31ea09e36f9eb5983 not found: ID does not exist"
Oct 30 19:54:56.560856 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.560775    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"]
Oct 30 19:54:56.568952 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:56.568885    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-426bab-f74447bc4-4khr2"]
Oct 30 19:54:57.208405 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:57.206984    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b59ec827-e83b-4e1f-b138-bccae9cc5356" path="/var/lib/kubelet/pods/b59ec827-e83b-4e1f-b138-bccae9cc5356/volumes"
Oct 30 19:54:58.441073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:58.437216    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:58.573672 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:58.573336    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5" podUID="2db52234-3bbf-4a48-b905-dec230b1b3f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:58.580233 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:58.580192    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:58.647193 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:58.645094    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 19:54:58.647193 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:58.645512    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:54:58.682210 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:58.682130    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": context deadline exceeded"
Oct 30 19:54:58.725288 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:58.725179    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:58.820277 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:58.820228    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r" podUID="3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 19:54:59.102213 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.102119    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:59.130590 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.130522    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:59.186213 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.186170    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:54:59.203300 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.203261    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:59.233429 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.233386    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:59.265271 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.265225    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:59.295213 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.295165    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:59.336231 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.335977    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded"
Oct 30 19:54:59.337350 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.336963    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5" podUID="2db52234-3bbf-4a48-b905-dec230b1b3f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 19:54:59.447180 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.447130    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:59.448140 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.448116    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:54:59.589073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.588162    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:59.593141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.589282    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:54:59.674519 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.646988    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 19:54:59.674519 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.647403    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:54:59.674519 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.647986    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:54:59.674519 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.648067    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:54:59.846117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.828196    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r" podUID="3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:59.846117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.829884    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:54:59.846117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.831239    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 19:54:59.874980 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.874902    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:54:59.913003 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.912068    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:54:59.913003 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.912927    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:54:59.921796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:54:59.916468    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:55:02.886812 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:02.886773    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"]
Oct 30 19:55:02.887405 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:02.887376    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb" podUID="2170ff4d-355b-4c88-ad37-cfd014e9bb17" containerName="maas-api" containerID="cri-o://5b98b7a86280e0793baede3cf76ed20102042ef2942e85fbcf0e3f6d6a37590a" gracePeriod=30
Oct 30 19:55:03.597094 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.595358    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:55:03.675408 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.675372    6030 generic.go:356] "Generic (PLEG): container finished" podID="2170ff4d-355b-4c88-ad37-cfd014e9bb17" containerID="5b98b7a86280e0793baede3cf76ed20102042ef2942e85fbcf0e3f6d6a37590a" exitCode=0
Oct 30 19:55:03.675600 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.675424    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"
Oct 30 19:55:03.675600 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.675464    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb" event={"ID":"2170ff4d-355b-4c88-ad37-cfd014e9bb17","Type":"ContainerDied","Data":"5b98b7a86280e0793baede3cf76ed20102042ef2942e85fbcf0e3f6d6a37590a"}
Oct 30 19:55:03.675600 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.675538    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb" event={"ID":"2170ff4d-355b-4c88-ad37-cfd014e9bb17","Type":"ContainerDied","Data":"b6a2fdd8427acd9e63127bd29268b1bfcd7eb0381ede74d2ffbf312964281b5f"}
Oct 30 19:55:03.675600 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.675571    6030 scope.go:122] "RemoveContainer" containerID="5b98b7a86280e0793baede3cf76ed20102042ef2942e85fbcf0e3f6d6a37590a"
Oct 30 19:55:03.685976 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.685955    6030 scope.go:122] "RemoveContainer" containerID="5b98b7a86280e0793baede3cf76ed20102042ef2942e85fbcf0e3f6d6a37590a"
Oct 30 19:55:03.686378 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:55:03.686356    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b98b7a86280e0793baede3cf76ed20102042ef2942e85fbcf0e3f6d6a37590a\": container with ID starting with 5b98b7a86280e0793baede3cf76ed20102042ef2942e85fbcf0e3f6d6a37590a not found: ID does not exist" containerID="5b98b7a86280e0793baede3cf76ed20102042ef2942e85fbcf0e3f6d6a37590a"
Oct 30 19:55:03.686467 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.686391    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b98b7a86280e0793baede3cf76ed20102042ef2942e85fbcf0e3f6d6a37590a"} err="failed to get container status \"5b98b7a86280e0793baede3cf76ed20102042ef2942e85fbcf0e3f6d6a37590a\": rpc error: code = NotFound desc = could not find container \"5b98b7a86280e0793baede3cf76ed20102042ef2942e85fbcf0e3f6d6a37590a\": container with ID starting with 5b98b7a86280e0793baede3cf76ed20102042ef2942e85fbcf0e3f6d6a37590a not found: ID does not exist"
Oct 30 19:55:03.698423 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.698395    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2170ff4d-355b-4c88-ad37-cfd014e9bb17-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2170ff4d-355b-4c88-ad37-cfd014e9bb17-maas-api-tls\") pod \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\" (UID: \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\") "
Oct 30 19:55:03.698568 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.698540    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2170ff4d-355b-4c88-ad37-cfd014e9bb17-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2170ff4d-355b-4c88-ad37-cfd014e9bb17-metrics-certs\") pod \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\" (UID: \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\") "
Oct 30 19:55:03.698642 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.698624    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2170ff4d-355b-4c88-ad37-cfd014e9bb17-kube-api-access-ljrj5\" (UniqueName: \"kubernetes.io/projected/2170ff4d-355b-4c88-ad37-cfd014e9bb17-kube-api-access-ljrj5\") pod \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\" (UID: \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\") "
Oct 30 19:55:03.698740 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.698705    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2170ff4d-355b-4c88-ad37-cfd014e9bb17-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2170ff4d-355b-4c88-ad37-cfd014e9bb17-openshift-service-ca\") pod \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\" (UID: \"2170ff4d-355b-4c88-ad37-cfd014e9bb17\") "
Oct 30 19:55:03.699132 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.699105    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2170ff4d-355b-4c88-ad37-cfd014e9bb17-openshift-service-ca" pod "2170ff4d-355b-4c88-ad37-cfd014e9bb17" (UID: "2170ff4d-355b-4c88-ad37-cfd014e9bb17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:03.699345 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.699328    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2170ff4d-355b-4c88-ad37-cfd014e9bb17-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:03.700892 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.700872    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2170ff4d-355b-4c88-ad37-cfd014e9bb17-metrics-certs" pod "2170ff4d-355b-4c88-ad37-cfd014e9bb17" (UID: "2170ff4d-355b-4c88-ad37-cfd014e9bb17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:03.700970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.700947    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2170ff4d-355b-4c88-ad37-cfd014e9bb17-kube-api-access-ljrj5" pod "2170ff4d-355b-4c88-ad37-cfd014e9bb17" (UID: "2170ff4d-355b-4c88-ad37-cfd014e9bb17"). InnerVolumeSpecName "kube-api-access-ljrj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:03.701235 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.701218    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2170ff4d-355b-4c88-ad37-cfd014e9bb17-maas-api-tls" pod "2170ff4d-355b-4c88-ad37-cfd014e9bb17" (UID: "2170ff4d-355b-4c88-ad37-cfd014e9bb17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:03.800796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.800714    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2170ff4d-355b-4c88-ad37-cfd014e9bb17-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:03.800796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.800742    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2170ff4d-355b-4c88-ad37-cfd014e9bb17-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:03.800796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:03.800753    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljrj5\" (UniqueName: \"kubernetes.io/projected/2170ff4d-355b-4c88-ad37-cfd014e9bb17-kube-api-access-ljrj5\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:04.002523 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:04.002476    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"]
Oct 30 19:55:04.005976 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:04.005952    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b17e94-69f66f9ccf-f48lb"]
Oct 30 19:55:04.289129 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:55:04.289089    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:55:05.208382 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:05.208347    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2170ff4d-355b-4c88-ad37-cfd014e9bb17" path="/var/lib/kubelet/pods/2170ff4d-355b-4c88-ad37-cfd014e9bb17/volumes"
Oct 30 19:55:13.095284 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:13.095236    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"]
Oct 30 19:55:13.095819 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:13.095491    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4" podUID="346006b7-21cc-469c-b8db-f82278c28c99" containerName="istio-proxy" containerID="cri-o://a0178749bf7bf8543f8d30b9125c08f118ec049576388b53b5833f89b75c0e28" gracePeriod=30
Oct 30 19:55:18.419069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.410291    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:55:18.495718 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.495683    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-workload-socket\") pod \"346006b7-21cc-469c-b8db-f82278c28c99\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") "
Oct 30 19:55:18.495905 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.495731    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-workload-certs\") pod \"346006b7-21cc-469c-b8db-f82278c28c99\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") "
Oct 30 19:55:18.495905 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.495787    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-istio-envoy\") pod \"346006b7-21cc-469c-b8db-f82278c28c99\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") "
Oct 30 19:55:18.495905 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.495847    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/346006b7-21cc-469c-b8db-f82278c28c99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/346006b7-21cc-469c-b8db-f82278c28c99-istio-podinfo\") pod \"346006b7-21cc-469c-b8db-f82278c28c99\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") "
Oct 30 19:55:18.495905 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.495885    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/346006b7-21cc-469c-b8db-f82278c28c99-kube-api-access-sgfhz\" (UniqueName: \"kubernetes.io/projected/346006b7-21cc-469c-b8db-f82278c28c99-kube-api-access-sgfhz\") pod \"346006b7-21cc-469c-b8db-f82278c28c99\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") "
Oct 30 19:55:18.496139 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.495944    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/346006b7-21cc-469c-b8db-f82278c28c99-istio-token\" (UniqueName: \"kubernetes.io/projected/346006b7-21cc-469c-b8db-f82278c28c99-istio-token\") pod \"346006b7-21cc-469c-b8db-f82278c28c99\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") "
Oct 30 19:55:18.496139 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.496002    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-workload-socket" pod "346006b7-21cc-469c-b8db-f82278c28c99" (UID: "346006b7-21cc-469c-b8db-f82278c28c99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:18.496139 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.496086    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-workload-certs" pod "346006b7-21cc-469c-b8db-f82278c28c99" (UID: "346006b7-21cc-469c-b8db-f82278c28c99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:18.496251 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.496177    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/346006b7-21cc-469c-b8db-f82278c28c99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/346006b7-21cc-469c-b8db-f82278c28c99-istiod-ca-cert\") pod \"346006b7-21cc-469c-b8db-f82278c28c99\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") "
Oct 30 19:55:18.496314 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.496288    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-istio-data\") pod \"346006b7-21cc-469c-b8db-f82278c28c99\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") "
Oct 30 19:55:18.496499 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.496353    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-credential-socket\") pod \"346006b7-21cc-469c-b8db-f82278c28c99\" (UID: \"346006b7-21cc-469c-b8db-f82278c28c99\") "
Oct 30 19:55:18.496573 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.496519    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-istio-data" pod "346006b7-21cc-469c-b8db-f82278c28c99" (UID: "346006b7-21cc-469c-b8db-f82278c28c99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:18.496645 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.496626    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/346006b7-21cc-469c-b8db-f82278c28c99-istiod-ca-cert" pod "346006b7-21cc-469c-b8db-f82278c28c99" (UID: "346006b7-21cc-469c-b8db-f82278c28c99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:18.496645 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.496630    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-credential-socket" pod "346006b7-21cc-469c-b8db-f82278c28c99" (UID: "346006b7-21cc-469c-b8db-f82278c28c99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:18.497190 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.497167    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/346006b7-21cc-469c-b8db-f82278c28c99-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:18.497293 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.497191    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:18.497293 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.497207    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:18.497293 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.497221    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:18.497293 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.497249    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:18.498761 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.498713    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-istio-envoy" pod "346006b7-21cc-469c-b8db-f82278c28c99" (UID: "346006b7-21cc-469c-b8db-f82278c28c99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:18.499132 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.499105    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/346006b7-21cc-469c-b8db-f82278c28c99-kube-api-access-sgfhz" pod "346006b7-21cc-469c-b8db-f82278c28c99" (UID: "346006b7-21cc-469c-b8db-f82278c28c99"). InnerVolumeSpecName "kube-api-access-sgfhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:18.500102 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.500063    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/346006b7-21cc-469c-b8db-f82278c28c99-istio-podinfo" pod "346006b7-21cc-469c-b8db-f82278c28c99" (UID: "346006b7-21cc-469c-b8db-f82278c28c99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:55:18.500102 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.500079    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/346006b7-21cc-469c-b8db-f82278c28c99-istio-token" pod "346006b7-21cc-469c-b8db-f82278c28c99" (UID: "346006b7-21cc-469c-b8db-f82278c28c99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:18.598280 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.598250    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/346006b7-21cc-469c-b8db-f82278c28c99-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:18.598280 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.598276    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/346006b7-21cc-469c-b8db-f82278c28c99-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:18.598280 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.598288    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgfhz\" (UniqueName: \"kubernetes.io/projected/346006b7-21cc-469c-b8db-f82278c28c99-kube-api-access-sgfhz\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:18.598549 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.598300    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/346006b7-21cc-469c-b8db-f82278c28c99-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:18.822957 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.822868    6030 generic.go:356] "Generic (PLEG): container finished" podID="346006b7-21cc-469c-b8db-f82278c28c99" containerID="a0178749bf7bf8543f8d30b9125c08f118ec049576388b53b5833f89b75c0e28" exitCode=0
Oct 30 19:55:18.822957 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.822925    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"
Oct 30 19:55:18.823183 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.822956    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4" event={"ID":"346006b7-21cc-469c-b8db-f82278c28c99","Type":"ContainerDied","Data":"a0178749bf7bf8543f8d30b9125c08f118ec049576388b53b5833f89b75c0e28"}
Oct 30 19:55:18.823183 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.823031    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4" event={"ID":"346006b7-21cc-469c-b8db-f82278c28c99","Type":"ContainerDied","Data":"e7adbd4d04f44fd2f101e0ef9987440d119d4d33da57be7fb6e9dbb7db675611"}
Oct 30 19:55:18.823183 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.823089    6030 scope.go:122] "RemoveContainer" containerID="a0178749bf7bf8543f8d30b9125c08f118ec049576388b53b5833f89b75c0e28"
Oct 30 19:55:18.833870 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.833849    6030 scope.go:122] "RemoveContainer" containerID="a0178749bf7bf8543f8d30b9125c08f118ec049576388b53b5833f89b75c0e28"
Oct 30 19:55:18.834183 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:55:18.834159    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0178749bf7bf8543f8d30b9125c08f118ec049576388b53b5833f89b75c0e28\": container with ID starting with a0178749bf7bf8543f8d30b9125c08f118ec049576388b53b5833f89b75c0e28 not found: ID does not exist" containerID="a0178749bf7bf8543f8d30b9125c08f118ec049576388b53b5833f89b75c0e28"
Oct 30 19:55:18.834294 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.834190    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0178749bf7bf8543f8d30b9125c08f118ec049576388b53b5833f89b75c0e28"} err="failed to get container status \"a0178749bf7bf8543f8d30b9125c08f118ec049576388b53b5833f89b75c0e28\": rpc error: code = NotFound desc = could not find container \"a0178749bf7bf8543f8d30b9125c08f118ec049576388b53b5833f89b75c0e28\": container with ID starting with a0178749bf7bf8543f8d30b9125c08f118ec049576388b53b5833f89b75c0e28 not found: ID does not exist"
Oct 30 19:55:18.851970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.851948    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"]
Oct 30 19:55:18.861904 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:18.861877    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-j9sq4"]
Oct 30 19:55:19.176280 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:19.176250    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"]
Oct 30 19:55:19.176504 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:19.176479    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6" podUID="39855966-df67-4ccf-87fe-c5ec63f95cdf" containerName="istio-proxy" containerID="cri-o://ee624a0ef73920fd1f28bae3a91371d753924f1666a945de84a0cd23987a37ba" gracePeriod=30
Oct 30 19:55:19.216685 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:19.216653    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="346006b7-21cc-469c-b8db-f82278c28c99" path="/var/lib/kubelet/pods/346006b7-21cc-469c-b8db-f82278c28c99/volumes"
Oct 30 19:55:24.507823 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.507795    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:55:24.609318 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.609285    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-data\") pod \"39855966-df67-4ccf-87fe-c5ec63f95cdf\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") "
Oct 30 19:55:24.609588 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.609569    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-token\" (UniqueName: \"kubernetes.io/projected/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-token\") pod \"39855966-df67-4ccf-87fe-c5ec63f95cdf\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") "
Oct 30 19:55:24.609721 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.609702    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-credential-socket\") pod \"39855966-df67-4ccf-87fe-c5ec63f95cdf\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") "
Oct 30 19:55:24.609892 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.609878    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-podinfo\") pod \"39855966-df67-4ccf-87fe-c5ec63f95cdf\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") "
Oct 30 19:55:24.610408 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.609988    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-data" pod "39855966-df67-4ccf-87fe-c5ec63f95cdf" (UID: "39855966-df67-4ccf-87fe-c5ec63f95cdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:24.610691 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.610669    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-envoy\") pod \"39855966-df67-4ccf-87fe-c5ec63f95cdf\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") "
Oct 30 19:55:24.610821 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.610748    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-workload-socket\") pod \"39855966-df67-4ccf-87fe-c5ec63f95cdf\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") "
Oct 30 19:55:24.610821 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.610809    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39855966-df67-4ccf-87fe-c5ec63f95cdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39855966-df67-4ccf-87fe-c5ec63f95cdf-istiod-ca-cert\") pod \"39855966-df67-4ccf-87fe-c5ec63f95cdf\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") "
Oct 30 19:55:24.610940 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.610868    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-workload-certs\") pod \"39855966-df67-4ccf-87fe-c5ec63f95cdf\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") "
Oct 30 19:55:24.610940 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.610931    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39855966-df67-4ccf-87fe-c5ec63f95cdf-kube-api-access-j57mw\" (UniqueName: \"kubernetes.io/projected/39855966-df67-4ccf-87fe-c5ec63f95cdf-kube-api-access-j57mw\") pod \"39855966-df67-4ccf-87fe-c5ec63f95cdf\" (UID: \"39855966-df67-4ccf-87fe-c5ec63f95cdf\") "
Oct 30 19:55:24.611707 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.611356    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-workload-socket" pod "39855966-df67-4ccf-87fe-c5ec63f95cdf" (UID: "39855966-df67-4ccf-87fe-c5ec63f95cdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:24.611707 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.611587    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-workload-certs" pod "39855966-df67-4ccf-87fe-c5ec63f95cdf" (UID: "39855966-df67-4ccf-87fe-c5ec63f95cdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:24.611707 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.611660    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:24.611707 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.611680    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:24.612140 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.612116    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-credential-socket" pod "39855966-df67-4ccf-87fe-c5ec63f95cdf" (UID: "39855966-df67-4ccf-87fe-c5ec63f95cdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:24.619088 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.619031    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-token" pod "39855966-df67-4ccf-87fe-c5ec63f95cdf" (UID: "39855966-df67-4ccf-87fe-c5ec63f95cdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:24.621328 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.621303    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39855966-df67-4ccf-87fe-c5ec63f95cdf-istiod-ca-cert" pod "39855966-df67-4ccf-87fe-c5ec63f95cdf" (UID: "39855966-df67-4ccf-87fe-c5ec63f95cdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:24.631212 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.631174    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-envoy" pod "39855966-df67-4ccf-87fe-c5ec63f95cdf" (UID: "39855966-df67-4ccf-87fe-c5ec63f95cdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:24.631665 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.631644    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-podinfo" pod "39855966-df67-4ccf-87fe-c5ec63f95cdf" (UID: "39855966-df67-4ccf-87fe-c5ec63f95cdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:55:24.631880 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.631833    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39855966-df67-4ccf-87fe-c5ec63f95cdf-kube-api-access-j57mw" pod "39855966-df67-4ccf-87fe-c5ec63f95cdf" (UID: "39855966-df67-4ccf-87fe-c5ec63f95cdf"). InnerVolumeSpecName "kube-api-access-j57mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:24.648809 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.648779    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"]
Oct 30 19:55:24.649168 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.649135    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w" podUID="0e9188e0-7232-4f19-8d67-82758f1db85c" containerName="payload-processing" containerID="cri-o://220177dd98bd2800306d0febcd573c94c06d4a51dfa31041fb77a46dce916d1d" gracePeriod=30
Oct 30 19:55:24.651732 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.651603    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"]
Oct 30 19:55:24.651874 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.651817    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx" podUID="273ea5ae-d8b5-4f08-b2e2-f7deb26def92" containerName="payload-pre-processing" containerID="cri-o://4e2a89fa64e0611e81f709fd00eba87f4729b3ef9d29701981c3ee0459633bc0" gracePeriod=30
Oct 30 19:55:24.712945 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.712918    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:24.713106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.712948    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:24.713106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.712963    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:24.713106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.712977    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:24.713106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.712991    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39855966-df67-4ccf-87fe-c5ec63f95cdf-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:24.713106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.713012    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39855966-df67-4ccf-87fe-c5ec63f95cdf-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:24.713106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.713026    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j57mw\" (UniqueName: \"kubernetes.io/projected/39855966-df67-4ccf-87fe-c5ec63f95cdf-kube-api-access-j57mw\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:24.871432 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.871403    6030 generic.go:356] "Generic (PLEG): container finished" podID="39855966-df67-4ccf-87fe-c5ec63f95cdf" containerID="ee624a0ef73920fd1f28bae3a91371d753924f1666a945de84a0cd23987a37ba" exitCode=0
Oct 30 19:55:24.871533 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.871494    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6" event={"ID":"39855966-df67-4ccf-87fe-c5ec63f95cdf","Type":"ContainerDied","Data":"ee624a0ef73920fd1f28bae3a91371d753924f1666a945de84a0cd23987a37ba"}
Oct 30 19:55:24.871587 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.871533    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6" event={"ID":"39855966-df67-4ccf-87fe-c5ec63f95cdf","Type":"ContainerDied","Data":"f7ea23e73f746fe3f23a05b912c289a48bdbab3b436cf862125ca5f3617ba5dd"}
Oct 30 19:55:24.871587 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.871555    6030 scope.go:122] "RemoveContainer" containerID="ee624a0ef73920fd1f28bae3a91371d753924f1666a945de84a0cd23987a37ba"
Oct 30 19:55:24.871693 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.871670    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"
Oct 30 19:55:24.899709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.898643    6030 generic.go:356] "Generic (PLEG): container finished" podID="0e9188e0-7232-4f19-8d67-82758f1db85c" containerID="220177dd98bd2800306d0febcd573c94c06d4a51dfa31041fb77a46dce916d1d" exitCode=0
Oct 30 19:55:24.899709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.898836    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w" event={"ID":"0e9188e0-7232-4f19-8d67-82758f1db85c","Type":"ContainerDied","Data":"220177dd98bd2800306d0febcd573c94c06d4a51dfa31041fb77a46dce916d1d"}
Oct 30 19:55:24.901893 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.901862    6030 generic.go:356] "Generic (PLEG): container finished" podID="273ea5ae-d8b5-4f08-b2e2-f7deb26def92" containerID="4e2a89fa64e0611e81f709fd00eba87f4729b3ef9d29701981c3ee0459633bc0" exitCode=0
Oct 30 19:55:24.902022 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.901905    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx" event={"ID":"273ea5ae-d8b5-4f08-b2e2-f7deb26def92","Type":"ContainerDied","Data":"4e2a89fa64e0611e81f709fd00eba87f4729b3ef9d29701981c3ee0459633bc0"}
Oct 30 19:55:24.915102 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.914274    6030 scope.go:122] "RemoveContainer" containerID="ee624a0ef73920fd1f28bae3a91371d753924f1666a945de84a0cd23987a37ba"
Oct 30 19:55:24.915344 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:55:24.915319    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee624a0ef73920fd1f28bae3a91371d753924f1666a945de84a0cd23987a37ba\": container with ID starting with ee624a0ef73920fd1f28bae3a91371d753924f1666a945de84a0cd23987a37ba not found: ID does not exist" containerID="ee624a0ef73920fd1f28bae3a91371d753924f1666a945de84a0cd23987a37ba"
Oct 30 19:55:24.915443 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.915358    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee624a0ef73920fd1f28bae3a91371d753924f1666a945de84a0cd23987a37ba"} err="failed to get container status \"ee624a0ef73920fd1f28bae3a91371d753924f1666a945de84a0cd23987a37ba\": rpc error: code = NotFound desc = could not find container \"ee624a0ef73920fd1f28bae3a91371d753924f1666a945de84a0cd23987a37ba\": container with ID starting with ee624a0ef73920fd1f28bae3a91371d753924f1666a945de84a0cd23987a37ba not found: ID does not exist"
Oct 30 19:55:24.936450 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.936398    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"]
Oct 30 19:55:24.940456 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.940413    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ldmb6"]
Oct 30 19:55:24.984890 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:24.984853    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"
Oct 30 19:55:25.024545 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.024479    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"
Oct 30 19:55:25.217071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.210787    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9188e0-7232-4f19-8d67-82758f1db85c-kube-api-access-btnmc\" (UniqueName: \"kubernetes.io/projected/0e9188e0-7232-4f19-8d67-82758f1db85c-kube-api-access-btnmc\") pod \"0e9188e0-7232-4f19-8d67-82758f1db85c\" (UID: \"0e9188e0-7232-4f19-8d67-82758f1db85c\") "
Oct 30 19:55:25.217071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.210861    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273ea5ae-d8b5-4f08-b2e2-f7deb26def92-kube-api-access-9zpwt\" (UniqueName: \"kubernetes.io/projected/273ea5ae-d8b5-4f08-b2e2-f7deb26def92-kube-api-access-9zpwt\") pod \"273ea5ae-d8b5-4f08-b2e2-f7deb26def92\" (UID: \"273ea5ae-d8b5-4f08-b2e2-f7deb26def92\") "
Oct 30 19:55:25.217071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.210951    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/273ea5ae-d8b5-4f08-b2e2-f7deb26def92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273ea5ae-d8b5-4f08-b2e2-f7deb26def92-plugins-config-volume\") pod \"273ea5ae-d8b5-4f08-b2e2-f7deb26def92\" (UID: \"273ea5ae-d8b5-4f08-b2e2-f7deb26def92\") "
Oct 30 19:55:25.217071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.211090    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9188e0-7232-4f19-8d67-82758f1db85c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9188e0-7232-4f19-8d67-82758f1db85c-plugins-config-volume\") pod \"0e9188e0-7232-4f19-8d67-82758f1db85c\" (UID: \"0e9188e0-7232-4f19-8d67-82758f1db85c\") "
Oct 30 19:55:25.243475 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.243411    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273ea5ae-d8b5-4f08-b2e2-f7deb26def92-plugins-config-volume" pod "273ea5ae-d8b5-4f08-b2e2-f7deb26def92" (UID: "273ea5ae-d8b5-4f08-b2e2-f7deb26def92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:25.334513 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.250095    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9188e0-7232-4f19-8d67-82758f1db85c-plugins-config-volume" pod "0e9188e0-7232-4f19-8d67-82758f1db85c" (UID: "0e9188e0-7232-4f19-8d67-82758f1db85c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:25.366319 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.356267    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9188e0-7232-4f19-8d67-82758f1db85c-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:25.366319 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.360132    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273ea5ae-d8b5-4f08-b2e2-f7deb26def92-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:25.372443 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.371779    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273ea5ae-d8b5-4f08-b2e2-f7deb26def92-kube-api-access-9zpwt" pod "273ea5ae-d8b5-4f08-b2e2-f7deb26def92" (UID: "273ea5ae-d8b5-4f08-b2e2-f7deb26def92"). InnerVolumeSpecName "kube-api-access-9zpwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:25.378985 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.378956    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39855966-df67-4ccf-87fe-c5ec63f95cdf" path="/var/lib/kubelet/pods/39855966-df67-4ccf-87fe-c5ec63f95cdf/volumes"
Oct 30 19:55:25.387566 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.387281    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9188e0-7232-4f19-8d67-82758f1db85c-kube-api-access-btnmc" pod "0e9188e0-7232-4f19-8d67-82758f1db85c" (UID: "0e9188e0-7232-4f19-8d67-82758f1db85c"). InnerVolumeSpecName "kube-api-access-btnmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:25.461837 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.461800    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btnmc\" (UniqueName: \"kubernetes.io/projected/0e9188e0-7232-4f19-8d67-82758f1db85c-kube-api-access-btnmc\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:25.461837 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.461843    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zpwt\" (UniqueName: \"kubernetes.io/projected/273ea5ae-d8b5-4f08-b2e2-f7deb26def92-kube-api-access-9zpwt\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:25.917148 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.917109    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w" event={"ID":"0e9188e0-7232-4f19-8d67-82758f1db85c","Type":"ContainerDied","Data":"d012399913b9ce806f1de797479ca6b1fbb4fe4fb7fd2d8b48c9b4854bcbbb91"}
Oct 30 19:55:25.917741 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.917720    6030 scope.go:122] "RemoveContainer" containerID="220177dd98bd2800306d0febcd573c94c06d4a51dfa31041fb77a46dce916d1d"
Oct 30 19:55:25.917991 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.917974    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"
Oct 30 19:55:25.943558 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.937754    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx" event={"ID":"273ea5ae-d8b5-4f08-b2e2-f7deb26def92","Type":"ContainerDied","Data":"e9694c67ee96d505d281335536125bf458e61bc5f7b148ee305e46bed15952de"}
Oct 30 19:55:25.943558 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.937865    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"
Oct 30 19:55:26.000223 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:25.999748    6030 scope.go:122] "RemoveContainer" containerID="4e2a89fa64e0611e81f709fd00eba87f4729b3ef9d29701981c3ee0459633bc0"
Oct 30 19:55:26.038311 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:26.038277    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"]
Oct 30 19:55:26.040781 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:26.040753    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5c95d4c796-pzh5w"]
Oct 30 19:55:26.078405 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:26.078373    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"]
Oct 30 19:55:26.089338 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:26.089298    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-84d4644b6c-m2tkx"]
Oct 30 19:55:26.975770 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:26.975211    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:27.218399 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:27.218365    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9188e0-7232-4f19-8d67-82758f1db85c" path="/var/lib/kubelet/pods/0e9188e0-7232-4f19-8d67-82758f1db85c/volumes"
Oct 30 19:55:27.219237 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:27.219214    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="273ea5ae-d8b5-4f08-b2e2-f7deb26def92" path="/var/lib/kubelet/pods/273ea5ae-d8b5-4f08-b2e2-f7deb26def92/volumes"
Oct 30 19:55:27.235141 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:27.232619    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:55:27.265160 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:27.265115    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:27.298402 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:27.297356    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:34.289539 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:55:34.289500    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:55:35.719231 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:35.719191    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"]
Oct 30 19:55:35.733972 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:35.733927    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"]
Oct 30 19:55:37.810346 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.810307    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"]
Oct 30 19:55:37.810997 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.810978    6030 cpu_manager.go:414] "containerMap: removing container" podUID="0e9188e0-7232-4f19-8d67-82758f1db85c" containerName="payload-processing"
Oct 30 19:55:37.810997 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.810998    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9188e0-7232-4f19-8d67-82758f1db85c" containerName="payload-processing"
Oct 30 19:55:37.811215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.811017    6030 cpu_manager.go:414] "containerMap: removing container" podUID="b59ec827-e83b-4e1f-b138-bccae9cc5356" containerName="maas-api"
Oct 30 19:55:37.811215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.811022    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b59ec827-e83b-4e1f-b138-bccae9cc5356" containerName="maas-api"
Oct 30 19:55:37.811215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.811031    6030 cpu_manager.go:414] "containerMap: removing container" podUID="346006b7-21cc-469c-b8db-f82278c28c99" containerName="istio-proxy"
Oct 30 19:55:37.811215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.811036    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="346006b7-21cc-469c-b8db-f82278c28c99" containerName="istio-proxy"
Oct 30 19:55:37.811215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.811069    6030 cpu_manager.go:414] "containerMap: removing container" podUID="39855966-df67-4ccf-87fe-c5ec63f95cdf" containerName="istio-proxy"
Oct 30 19:55:37.811215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.811077    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39855966-df67-4ccf-87fe-c5ec63f95cdf" containerName="istio-proxy"
Oct 30 19:55:37.811215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.811085    6030 cpu_manager.go:414] "containerMap: removing container" podUID="2170ff4d-355b-4c88-ad37-cfd014e9bb17" containerName="maas-api"
Oct 30 19:55:37.811215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.811090    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2170ff4d-355b-4c88-ad37-cfd014e9bb17" containerName="maas-api"
Oct 30 19:55:37.811215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.811097    6030 cpu_manager.go:414] "containerMap: removing container" podUID="273ea5ae-d8b5-4f08-b2e2-f7deb26def92" containerName="payload-pre-processing"
Oct 30 19:55:37.811215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.811102    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273ea5ae-d8b5-4f08-b2e2-f7deb26def92" containerName="payload-pre-processing"
Oct 30 19:55:37.811215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.811207    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="b59ec827-e83b-4e1f-b138-bccae9cc5356" containerName="maas-api"
Oct 30 19:55:37.811215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.811217    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="346006b7-21cc-469c-b8db-f82278c28c99" containerName="istio-proxy"
Oct 30 19:55:37.811215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.811225    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="39855966-df67-4ccf-87fe-c5ec63f95cdf" containerName="istio-proxy"
Oct 30 19:55:37.811867 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.811232    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="2170ff4d-355b-4c88-ad37-cfd014e9bb17" containerName="maas-api"
Oct 30 19:55:37.811867 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.811239    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="273ea5ae-d8b5-4f08-b2e2-f7deb26def92" containerName="payload-pre-processing"
Oct 30 19:55:37.811867 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.811245    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9188e0-7232-4f19-8d67-82758f1db85c" containerName="payload-processing"
Oct 30 19:55:37.816157 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.816136    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.826653 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.826631    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-927bf8-openshift-default-dockercfg-kr48w\""
Oct 30 19:55:37.829625 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.829598    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"]
Oct 30 19:55:37.858052 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.858013    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29ff919-34f3-412f-adf8-d331698289e9-istiod-ca-cert\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.858285 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.858140    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-istio-data\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.858285 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.858189    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-workload-socket\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.858402 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.858298    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-workload-certs\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.858402 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.858345    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a29ff919-34f3-412f-adf8-d331698289e9-istio-token\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.858564 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.858476    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87xsd\" (UniqueName: \"kubernetes.io/projected/a29ff919-34f3-412f-adf8-d331698289e9-kube-api-access-87xsd\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.858564 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.858529    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-istio-envoy\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.858649 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.858583    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29ff919-34f3-412f-adf8-d331698289e9-istio-podinfo\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.858696 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.858673    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-credential-socket\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.967071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.961241    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29ff919-34f3-412f-adf8-d331698289e9-istiod-ca-cert\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.967071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.961315    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-istio-data\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.967071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.961361    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-workload-socket\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.967071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.961416    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-workload-certs\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.967071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.961453    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a29ff919-34f3-412f-adf8-d331698289e9-istio-token\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.967071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.961516    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87xsd\" (UniqueName: \"kubernetes.io/projected/a29ff919-34f3-412f-adf8-d331698289e9-kube-api-access-87xsd\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.967071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.961553    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-istio-envoy\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.967071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.961667    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29ff919-34f3-412f-adf8-d331698289e9-istio-podinfo\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.967071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.961746    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-credential-socket\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.967071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.962312    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-credential-socket\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.967071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.963118    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29ff919-34f3-412f-adf8-d331698289e9-istiod-ca-cert\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.967071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.963370    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-istio-data\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.967071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.963629    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-workload-socket\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:37.967071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:37.963877    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-workload-certs\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:38.000712 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:38.000680    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-istio-envoy\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:38.001440 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:38.001417    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29ff919-34f3-412f-adf8-d331698289e9-istio-podinfo\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:38.028719 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:38.028686    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87xsd\" (UniqueName: \"kubernetes.io/projected/a29ff919-34f3-412f-adf8-d331698289e9-kube-api-access-87xsd\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:38.061139 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:38.058761    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a29ff919-34f3-412f-adf8-d331698289e9-istio-token\") pod \"e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") " pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:38.134091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:38.134027    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:38.440075 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:38.440021    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"]
Oct 30 19:55:38.441581 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:55:38.441540    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29ff919_34f3_412f_adf8_d331698289e9.slice/crio-c55e66a939111b1e4b2f096eb22639a60bacfd97f84df1c97a8c8e3a3de28b3b WatchSource:0}: Error finding container c55e66a939111b1e4b2f096eb22639a60bacfd97f84df1c97a8c8e3a3de28b3b: Status 404 returned error can't find the container with id c55e66a939111b1e4b2f096eb22639a60bacfd97f84df1c97a8c8e3a3de28b3b
Oct 30 19:55:38.444668 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:38.444627    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:55:38.444770 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:38.444720    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:55:38.444850 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:38.444768    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:55:39.049872 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:39.049836    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz" event={"ID":"a29ff919-34f3-412f-adf8-d331698289e9","Type":"ContainerStarted","Data":"0c711057cba171197487995af0ab346ee6fbe181ca195025e5cece206601db4a"}
Oct 30 19:55:39.050417 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:39.049900    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz" event={"ID":"a29ff919-34f3-412f-adf8-d331698289e9","Type":"ContainerStarted","Data":"c55e66a939111b1e4b2f096eb22639a60bacfd97f84df1c97a8c8e3a3de28b3b"}
Oct 30 19:55:39.134236 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:39.134207    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:39.139845 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:39.139821    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:39.184823 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:39.184764    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz" podStartSLOduration=2.184750094 podStartE2EDuration="2.184750094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:39.093935119 +0000 UTC m=+1418.009541975" watchObservedRunningTime="2026-10-30 19:55:39.184750094 +0000 UTC m=+1418.100356949"
Oct 30 19:55:40.055999 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:40.055973    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:40.057132 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:40.057112    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:55:40.721219 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:40.721147    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp" podUID="d61e89eb-9e7b-413d-9558-2382b36f29d9" containerName="payload-processing" containerID="cri-o://2015f29ca555e14b476df269cbd8b3f1a0ab3a2d9ba490db66a5fb5e30282750" gracePeriod=25
Oct 30 19:55:40.734623 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:40.734585    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h" podUID="e9e770bf-b4c1-47c3-b850-92c9d42dd9ba" containerName="payload-pre-processing" containerID="cri-o://52cf17c020737c0f119752453489c10d5a541be633b29e1c4b85097935efd570" gracePeriod=25
Oct 30 19:55:41.020183 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.020157    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"
Oct 30 19:55:41.039766 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.039745    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"
Oct 30 19:55:41.064191 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.064160    6030 generic.go:356] "Generic (PLEG): container finished" podID="e9e770bf-b4c1-47c3-b850-92c9d42dd9ba" containerID="52cf17c020737c0f119752453489c10d5a541be633b29e1c4b85097935efd570" exitCode=0
Oct 30 19:55:41.064580 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.064197    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"
Oct 30 19:55:41.064580 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.064236    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h" event={"ID":"e9e770bf-b4c1-47c3-b850-92c9d42dd9ba","Type":"ContainerDied","Data":"52cf17c020737c0f119752453489c10d5a541be633b29e1c4b85097935efd570"}
Oct 30 19:55:41.064580 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.064318    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h" event={"ID":"e9e770bf-b4c1-47c3-b850-92c9d42dd9ba","Type":"ContainerDied","Data":"67f182bd03cded515b5134e96e9a3cb9d184050e19b72522f538f38cf1e880e6"}
Oct 30 19:55:41.064580 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.064330    6030 scope.go:122] "RemoveContainer" containerID="52cf17c020737c0f119752453489c10d5a541be633b29e1c4b85097935efd570"
Oct 30 19:55:41.065880 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.065860    6030 generic.go:356] "Generic (PLEG): container finished" podID="d61e89eb-9e7b-413d-9558-2382b36f29d9" containerID="2015f29ca555e14b476df269cbd8b3f1a0ab3a2d9ba490db66a5fb5e30282750" exitCode=0
Oct 30 19:55:41.065987 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.065912    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"
Oct 30 19:55:41.066064 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.065987    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp" event={"ID":"d61e89eb-9e7b-413d-9558-2382b36f29d9","Type":"ContainerDied","Data":"2015f29ca555e14b476df269cbd8b3f1a0ab3a2d9ba490db66a5fb5e30282750"}
Oct 30 19:55:41.066064 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.066030    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp" event={"ID":"d61e89eb-9e7b-413d-9558-2382b36f29d9","Type":"ContainerDied","Data":"8d103582bcad7e423b842e7a28541c146e0ada9afb4ca4b06b1bf19567e7f59f"}
Oct 30 19:55:41.076032 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.075926    6030 scope.go:122] "RemoveContainer" containerID="52cf17c020737c0f119752453489c10d5a541be633b29e1c4b85097935efd570"
Oct 30 19:55:41.076347 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:55:41.076319    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52cf17c020737c0f119752453489c10d5a541be633b29e1c4b85097935efd570\": container with ID starting with 52cf17c020737c0f119752453489c10d5a541be633b29e1c4b85097935efd570 not found: ID does not exist" containerID="52cf17c020737c0f119752453489c10d5a541be633b29e1c4b85097935efd570"
Oct 30 19:55:41.076461 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.076360    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52cf17c020737c0f119752453489c10d5a541be633b29e1c4b85097935efd570"} err="failed to get container status \"52cf17c020737c0f119752453489c10d5a541be633b29e1c4b85097935efd570\": rpc error: code = NotFound desc = could not find container \"52cf17c020737c0f119752453489c10d5a541be633b29e1c4b85097935efd570\": container with ID starting with 52cf17c020737c0f119752453489c10d5a541be633b29e1c4b85097935efd570 not found: ID does not exist"
Oct 30 19:55:41.076461 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.076385    6030 scope.go:122] "RemoveContainer" containerID="2015f29ca555e14b476df269cbd8b3f1a0ab3a2d9ba490db66a5fb5e30282750"
Oct 30 19:55:41.091539 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.091520    6030 scope.go:122] "RemoveContainer" containerID="2015f29ca555e14b476df269cbd8b3f1a0ab3a2d9ba490db66a5fb5e30282750"
Oct 30 19:55:41.091933 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:55:41.091903    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2015f29ca555e14b476df269cbd8b3f1a0ab3a2d9ba490db66a5fb5e30282750\": container with ID starting with 2015f29ca555e14b476df269cbd8b3f1a0ab3a2d9ba490db66a5fb5e30282750 not found: ID does not exist" containerID="2015f29ca555e14b476df269cbd8b3f1a0ab3a2d9ba490db66a5fb5e30282750"
Oct 30 19:55:41.092028 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.091942    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2015f29ca555e14b476df269cbd8b3f1a0ab3a2d9ba490db66a5fb5e30282750"} err="failed to get container status \"2015f29ca555e14b476df269cbd8b3f1a0ab3a2d9ba490db66a5fb5e30282750\": rpc error: code = NotFound desc = could not find container \"2015f29ca555e14b476df269cbd8b3f1a0ab3a2d9ba490db66a5fb5e30282750\": container with ID starting with 2015f29ca555e14b476df269cbd8b3f1a0ab3a2d9ba490db66a5fb5e30282750 not found: ID does not exist"
Oct 30 19:55:41.105087 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.105033    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e770bf-b4c1-47c3-b850-92c9d42dd9ba-kube-api-access-r9lxs\" (UniqueName: \"kubernetes.io/projected/e9e770bf-b4c1-47c3-b850-92c9d42dd9ba-kube-api-access-r9lxs\") pod \"e9e770bf-b4c1-47c3-b850-92c9d42dd9ba\" (UID: \"e9e770bf-b4c1-47c3-b850-92c9d42dd9ba\") "
Oct 30 19:55:41.105209 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.105192    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61e89eb-9e7b-413d-9558-2382b36f29d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61e89eb-9e7b-413d-9558-2382b36f29d9-plugins-config-volume\") pod \"d61e89eb-9e7b-413d-9558-2382b36f29d9\" (UID: \"d61e89eb-9e7b-413d-9558-2382b36f29d9\") "
Oct 30 19:55:41.105672 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.105308    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61e89eb-9e7b-413d-9558-2382b36f29d9-kube-api-access-m852k\" (UniqueName: \"kubernetes.io/projected/d61e89eb-9e7b-413d-9558-2382b36f29d9-kube-api-access-m852k\") pod \"d61e89eb-9e7b-413d-9558-2382b36f29d9\" (UID: \"d61e89eb-9e7b-413d-9558-2382b36f29d9\") "
Oct 30 19:55:41.105672 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.105343    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e770bf-b4c1-47c3-b850-92c9d42dd9ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e770bf-b4c1-47c3-b850-92c9d42dd9ba-plugins-config-volume\") pod \"e9e770bf-b4c1-47c3-b850-92c9d42dd9ba\" (UID: \"e9e770bf-b4c1-47c3-b850-92c9d42dd9ba\") "
Oct 30 19:55:41.105672 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.105637    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61e89eb-9e7b-413d-9558-2382b36f29d9-plugins-config-volume" pod "d61e89eb-9e7b-413d-9558-2382b36f29d9" (UID: "d61e89eb-9e7b-413d-9558-2382b36f29d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:41.105856 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.105753    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e770bf-b4c1-47c3-b850-92c9d42dd9ba-plugins-config-volume" pod "e9e770bf-b4c1-47c3-b850-92c9d42dd9ba" (UID: "e9e770bf-b4c1-47c3-b850-92c9d42dd9ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:41.105997 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.105977    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e770bf-b4c1-47c3-b850-92c9d42dd9ba-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:41.106076 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.106005    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61e89eb-9e7b-413d-9558-2382b36f29d9-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:41.109995 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.109965    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61e89eb-9e7b-413d-9558-2382b36f29d9-kube-api-access-m852k" pod "d61e89eb-9e7b-413d-9558-2382b36f29d9" (UID: "d61e89eb-9e7b-413d-9558-2382b36f29d9"). InnerVolumeSpecName "kube-api-access-m852k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:41.111090 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.111034    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e770bf-b4c1-47c3-b850-92c9d42dd9ba-kube-api-access-r9lxs" pod "e9e770bf-b4c1-47c3-b850-92c9d42dd9ba" (UID: "e9e770bf-b4c1-47c3-b850-92c9d42dd9ba"). InnerVolumeSpecName "kube-api-access-r9lxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:41.206956 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.206913    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m852k\" (UniqueName: \"kubernetes.io/projected/d61e89eb-9e7b-413d-9558-2382b36f29d9-kube-api-access-m852k\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:41.206956 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.206953    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9lxs\" (UniqueName: \"kubernetes.io/projected/e9e770bf-b4c1-47c3-b850-92c9d42dd9ba-kube-api-access-r9lxs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:41.407069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.406683    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"]
Oct 30 19:55:41.422505 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.422472    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d2dd78-5ffbd744f8-d8qxp"]
Oct 30 19:55:41.480662 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.480624    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"]
Oct 30 19:55:41.485019 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:41.484988    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d2dd78-7ff77c445f-59v4h"]
Oct 30 19:55:42.992200 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:42.992133    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"]
Oct 30 19:55:42.992677 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:42.992663    6030 cpu_manager.go:414] "containerMap: removing container" podUID="d61e89eb-9e7b-413d-9558-2382b36f29d9" containerName="payload-processing"
Oct 30 19:55:42.992718 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:42.992679    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61e89eb-9e7b-413d-9558-2382b36f29d9" containerName="payload-processing"
Oct 30 19:55:42.992718 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:42.992697    6030 cpu_manager.go:414] "containerMap: removing container" podUID="e9e770bf-b4c1-47c3-b850-92c9d42dd9ba" containerName="payload-pre-processing"
Oct 30 19:55:42.992718 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:42.992703    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e770bf-b4c1-47c3-b850-92c9d42dd9ba" containerName="payload-pre-processing"
Oct 30 19:55:42.992816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:42.992779    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61e89eb-9e7b-413d-9558-2382b36f29d9" containerName="payload-processing"
Oct 30 19:55:42.992816 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:42.992792    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e770bf-b4c1-47c3-b850-92c9d42dd9ba" containerName="payload-pre-processing"
Oct 30 19:55:42.997412 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:42.997394    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"
Oct 30 19:55:43.008058 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:43.008019    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-927bf8-dockercfg-jd75v\""
Oct 30 19:55:43.009554 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:43.009471    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-927bf8\""
Oct 30 19:55:43.019500 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:43.014365    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"]
Oct 30 19:55:43.051004 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:43.050968    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tffdt\" (UniqueName: \"kubernetes.io/projected/755da169-322e-451f-b4ac-90053aa2b956-kube-api-access-tffdt\") pod \"payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx\" (UID: \"755da169-322e-451f-b4ac-90053aa2b956\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"
Oct 30 19:55:43.051204 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:43.051189    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/755da169-322e-451f-b4ac-90053aa2b956-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx\" (UID: \"755da169-322e-451f-b4ac-90053aa2b956\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"
Oct 30 19:55:43.159068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:43.156817    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/755da169-322e-451f-b4ac-90053aa2b956-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx\" (UID: \"755da169-322e-451f-b4ac-90053aa2b956\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"
Oct 30 19:55:43.159068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:43.156936    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tffdt\" (UniqueName: \"kubernetes.io/projected/755da169-322e-451f-b4ac-90053aa2b956-kube-api-access-tffdt\") pod \"payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx\" (UID: \"755da169-322e-451f-b4ac-90053aa2b956\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"
Oct 30 19:55:43.159068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:43.158091    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/755da169-322e-451f-b4ac-90053aa2b956-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx\" (UID: \"755da169-322e-451f-b4ac-90053aa2b956\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"
Oct 30 19:55:43.190115 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:43.181074    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tffdt\" (UniqueName: \"kubernetes.io/projected/755da169-322e-451f-b4ac-90053aa2b956-kube-api-access-tffdt\") pod \"payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx\" (UID: \"755da169-322e-451f-b4ac-90053aa2b956\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"
Oct 30 19:55:43.215619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:43.215338    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61e89eb-9e7b-413d-9558-2382b36f29d9" path="/var/lib/kubelet/pods/d61e89eb-9e7b-413d-9558-2382b36f29d9/volumes"
Oct 30 19:55:43.215833 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:43.215802    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e770bf-b4c1-47c3-b850-92c9d42dd9ba" path="/var/lib/kubelet/pods/e9e770bf-b4c1-47c3-b850-92c9d42dd9ba/volumes"
Oct 30 19:55:43.232092 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:43.225188    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"]
Oct 30 19:55:43.232092 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:43.225439    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9" podUID="8c76eb36-5c29-41fa-8289-d8f08fba2181" containerName="istio-proxy" containerID="cri-o://eccd86168122e976ae0bf9ee144d4b6bb0ce462315fafd25d506b52f7f8712af" gracePeriod=30
Oct 30 19:55:43.312792 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:43.312720    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"
Oct 30 19:55:43.635464 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:43.635433    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"]
Oct 30 19:55:43.637105 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:55:43.637074    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755da169_322e_451f_b4ac_90053aa2b956.slice/crio-efa17639b0ccb55af3dcbae412fc6390dd5504886315551f4d96e82b8403dc21 WatchSource:0}: Error finding container efa17639b0ccb55af3dcbae412fc6390dd5504886315551f4d96e82b8403dc21: Status 404 returned error can't find the container with id efa17639b0ccb55af3dcbae412fc6390dd5504886315551f4d96e82b8403dc21
Oct 30 19:55:43.817509 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:43.817476    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"]
Oct 30 19:55:44.148337 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.135255    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk" podUID="d94e0001-0f96-4e02-bbba-fd81d82584e9" containerName="maas-api" containerID="cri-o://0699336a3b105d30880d4679a12a7e9595f2d4c8e6524cdd0c38680c2e67fdd7" gracePeriod=30
Oct 30 19:55:44.149627 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.148928    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx" event={"ID":"755da169-322e-451f-b4ac-90053aa2b956","Type":"ContainerStarted","Data":"5e8f0f8be5f0ef3befa5fd647e1cbb841fd1dbee7bad264ae65ebf6813325860"}
Oct 30 19:55:44.149627 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.149001    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx" event={"ID":"755da169-322e-451f-b4ac-90053aa2b956","Type":"ContainerStarted","Data":"efa17639b0ccb55af3dcbae412fc6390dd5504886315551f4d96e82b8403dc21"}
Oct 30 19:55:44.149627 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.149029    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"
Oct 30 19:55:44.215094 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.213401    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx" podStartSLOduration=2.213380355 podStartE2EDuration="2.213380355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:44.212999274 +0000 UTC m=+1423.128606131" watchObservedRunningTime="2026-10-30 19:55:44.213380355 +0000 UTC m=+1423.128987214"
Oct 30 19:55:44.289097 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.289029    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk" podUID="d94e0001-0f96-4e02-bbba-fd81d82584e9" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.119:8443/health\": dial tcp 10.133.0.119:8443: connect: connection refused"
Oct 30 19:55:44.476228 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.476197    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:55:44.585912 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.585883    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d94e0001-0f96-4e02-bbba-fd81d82584e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d94e0001-0f96-4e02-bbba-fd81d82584e9-maas-api-tls\") pod \"d94e0001-0f96-4e02-bbba-fd81d82584e9\" (UID: \"d94e0001-0f96-4e02-bbba-fd81d82584e9\") "
Oct 30 19:55:44.586120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.585924    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d94e0001-0f96-4e02-bbba-fd81d82584e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94e0001-0f96-4e02-bbba-fd81d82584e9-openshift-service-ca\") pod \"d94e0001-0f96-4e02-bbba-fd81d82584e9\" (UID: \"d94e0001-0f96-4e02-bbba-fd81d82584e9\") "
Oct 30 19:55:44.586120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.586059    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d94e0001-0f96-4e02-bbba-fd81d82584e9-kube-api-access-kplj7\" (UniqueName: \"kubernetes.io/projected/d94e0001-0f96-4e02-bbba-fd81d82584e9-kube-api-access-kplj7\") pod \"d94e0001-0f96-4e02-bbba-fd81d82584e9\" (UID: \"d94e0001-0f96-4e02-bbba-fd81d82584e9\") "
Oct 30 19:55:44.586120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.586113    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d94e0001-0f96-4e02-bbba-fd81d82584e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d94e0001-0f96-4e02-bbba-fd81d82584e9-metrics-certs\") pod \"d94e0001-0f96-4e02-bbba-fd81d82584e9\" (UID: \"d94e0001-0f96-4e02-bbba-fd81d82584e9\") "
Oct 30 19:55:44.586338 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.586316    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d94e0001-0f96-4e02-bbba-fd81d82584e9-openshift-service-ca" pod "d94e0001-0f96-4e02-bbba-fd81d82584e9" (UID: "d94e0001-0f96-4e02-bbba-fd81d82584e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:44.586577 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.586557    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94e0001-0f96-4e02-bbba-fd81d82584e9-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:44.588288 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.588263    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d94e0001-0f96-4e02-bbba-fd81d82584e9-maas-api-tls" pod "d94e0001-0f96-4e02-bbba-fd81d82584e9" (UID: "d94e0001-0f96-4e02-bbba-fd81d82584e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:44.588397 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.588296    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d94e0001-0f96-4e02-bbba-fd81d82584e9-metrics-certs" pod "d94e0001-0f96-4e02-bbba-fd81d82584e9" (UID: "d94e0001-0f96-4e02-bbba-fd81d82584e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:44.588397 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.588345    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d94e0001-0f96-4e02-bbba-fd81d82584e9-kube-api-access-kplj7" pod "d94e0001-0f96-4e02-bbba-fd81d82584e9" (UID: "d94e0001-0f96-4e02-bbba-fd81d82584e9"). InnerVolumeSpecName "kube-api-access-kplj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:44.697127 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.690091    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d94e0001-0f96-4e02-bbba-fd81d82584e9-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:44.697127 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.690131    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kplj7\" (UniqueName: \"kubernetes.io/projected/d94e0001-0f96-4e02-bbba-fd81d82584e9-kube-api-access-kplj7\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:44.697127 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:44.690146    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d94e0001-0f96-4e02-bbba-fd81d82584e9-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:45.142060 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:45.142016    6030 generic.go:356] "Generic (PLEG): container finished" podID="d94e0001-0f96-4e02-bbba-fd81d82584e9" containerID="0699336a3b105d30880d4679a12a7e9595f2d4c8e6524cdd0c38680c2e67fdd7" exitCode=0
Oct 30 19:55:45.142226 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:45.142075    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk" event={"ID":"d94e0001-0f96-4e02-bbba-fd81d82584e9","Type":"ContainerDied","Data":"0699336a3b105d30880d4679a12a7e9595f2d4c8e6524cdd0c38680c2e67fdd7"}
Oct 30 19:55:45.142226 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:45.142128    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk" event={"ID":"d94e0001-0f96-4e02-bbba-fd81d82584e9","Type":"ContainerDied","Data":"dff5e2f3e3a5c3100d7d7f641351b1c14503a7d2a54dd8a80e1bc8859eb4fbdf"}
Oct 30 19:55:45.142226 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:45.142084    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"
Oct 30 19:55:45.142226 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:45.142158    6030 scope.go:122] "RemoveContainer" containerID="0699336a3b105d30880d4679a12a7e9595f2d4c8e6524cdd0c38680c2e67fdd7"
Oct 30 19:55:45.152521 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:45.152501    6030 scope.go:122] "RemoveContainer" containerID="0699336a3b105d30880d4679a12a7e9595f2d4c8e6524cdd0c38680c2e67fdd7"
Oct 30 19:55:45.152836 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:55:45.152785    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0699336a3b105d30880d4679a12a7e9595f2d4c8e6524cdd0c38680c2e67fdd7\": container with ID starting with 0699336a3b105d30880d4679a12a7e9595f2d4c8e6524cdd0c38680c2e67fdd7 not found: ID does not exist" containerID="0699336a3b105d30880d4679a12a7e9595f2d4c8e6524cdd0c38680c2e67fdd7"
Oct 30 19:55:45.152836 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:45.152809    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0699336a3b105d30880d4679a12a7e9595f2d4c8e6524cdd0c38680c2e67fdd7"} err="failed to get container status \"0699336a3b105d30880d4679a12a7e9595f2d4c8e6524cdd0c38680c2e67fdd7\": rpc error: code = NotFound desc = could not find container \"0699336a3b105d30880d4679a12a7e9595f2d4c8e6524cdd0c38680c2e67fdd7\": container with ID starting with 0699336a3b105d30880d4679a12a7e9595f2d4c8e6524cdd0c38680c2e67fdd7 not found: ID does not exist"
Oct 30 19:55:45.167854 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:45.167832    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"]
Oct 30 19:55:45.174163 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:45.174134    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d2dd78-57445c5f8-pr5jk"]
Oct 30 19:55:45.202177 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:45.202150    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d94e0001-0f96-4e02-bbba-fd81d82584e9" path="/var/lib/kubelet/pods/d94e0001-0f96-4e02-bbba-fd81d82584e9/volumes"
Oct 30 19:55:48.511351 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.511328    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:55:48.551695 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.551612    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-data\") pod \"8c76eb36-5c29-41fa-8289-d8f08fba2181\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") "
Oct 30 19:55:48.551695 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.551680    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-credential-socket\") pod \"8c76eb36-5c29-41fa-8289-d8f08fba2181\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") "
Oct 30 19:55:48.551931 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.551743    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-token\" (UniqueName: \"kubernetes.io/projected/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-token\") pod \"8c76eb36-5c29-41fa-8289-d8f08fba2181\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") "
Oct 30 19:55:48.551931 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.551797    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-workload-socket\") pod \"8c76eb36-5c29-41fa-8289-d8f08fba2181\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") "
Oct 30 19:55:48.552062 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.551942    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-data" pod "8c76eb36-5c29-41fa-8289-d8f08fba2181" (UID: "8c76eb36-5c29-41fa-8289-d8f08fba2181"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:48.552062 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.551990    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-credential-socket" pod "8c76eb36-5c29-41fa-8289-d8f08fba2181" (UID: "8c76eb36-5c29-41fa-8289-d8f08fba2181"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:48.552204 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.552182    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c76eb36-5c29-41fa-8289-d8f08fba2181-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c76eb36-5c29-41fa-8289-d8f08fba2181-istiod-ca-cert\") pod \"8c76eb36-5c29-41fa-8289-d8f08fba2181\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") "
Oct 30 19:55:48.552596 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.552569    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-podinfo\") pod \"8c76eb36-5c29-41fa-8289-d8f08fba2181\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") "
Oct 30 19:55:48.552722 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.552659    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-envoy\") pod \"8c76eb36-5c29-41fa-8289-d8f08fba2181\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") "
Oct 30 19:55:48.552778 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.552723    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c76eb36-5c29-41fa-8289-d8f08fba2181-kube-api-access-r9bzn\" (UniqueName: \"kubernetes.io/projected/8c76eb36-5c29-41fa-8289-d8f08fba2181-kube-api-access-r9bzn\") pod \"8c76eb36-5c29-41fa-8289-d8f08fba2181\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") "
Oct 30 19:55:48.552778 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.552688    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-workload-socket" pod "8c76eb36-5c29-41fa-8289-d8f08fba2181" (UID: "8c76eb36-5c29-41fa-8289-d8f08fba2181"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:48.552877 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.552787    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-workload-certs\") pod \"8c76eb36-5c29-41fa-8289-d8f08fba2181\" (UID: \"8c76eb36-5c29-41fa-8289-d8f08fba2181\") "
Oct 30 19:55:48.552877 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.552857    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c76eb36-5c29-41fa-8289-d8f08fba2181-istiod-ca-cert" pod "8c76eb36-5c29-41fa-8289-d8f08fba2181" (UID: "8c76eb36-5c29-41fa-8289-d8f08fba2181"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:48.553418 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.553224    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-workload-certs" pod "8c76eb36-5c29-41fa-8289-d8f08fba2181" (UID: "8c76eb36-5c29-41fa-8289-d8f08fba2181"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:48.553418 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.553391    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.553418 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.553410    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.553595 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.553425    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.553595 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.553440    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.553595 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.553455    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c76eb36-5c29-41fa-8289-d8f08fba2181-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.561082 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.555654    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c76eb36-5c29-41fa-8289-d8f08fba2181-kube-api-access-r9bzn" pod "8c76eb36-5c29-41fa-8289-d8f08fba2181" (UID: "8c76eb36-5c29-41fa-8289-d8f08fba2181"). InnerVolumeSpecName "kube-api-access-r9bzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:48.561082 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.555842    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-token" pod "8c76eb36-5c29-41fa-8289-d8f08fba2181" (UID: "8c76eb36-5c29-41fa-8289-d8f08fba2181"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:48.561082 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.555908    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-envoy" pod "8c76eb36-5c29-41fa-8289-d8f08fba2181" (UID: "8c76eb36-5c29-41fa-8289-d8f08fba2181"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:48.561082 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.558175    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-podinfo" pod "8c76eb36-5c29-41fa-8289-d8f08fba2181" (UID: "8c76eb36-5c29-41fa-8289-d8f08fba2181"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:55:48.655278 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.655192    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.655278 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.655227    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.655278 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.655241    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c76eb36-5c29-41fa-8289-d8f08fba2181-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.655278 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:48.655254    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9bzn\" (UniqueName: \"kubernetes.io/projected/8c76eb36-5c29-41fa-8289-d8f08fba2181-kube-api-access-r9bzn\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:49.182071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:49.172002    6030 generic.go:356] "Generic (PLEG): container finished" podID="8c76eb36-5c29-41fa-8289-d8f08fba2181" containerID="eccd86168122e976ae0bf9ee144d4b6bb0ce462315fafd25d506b52f7f8712af" exitCode=0
Oct 30 19:55:49.182071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:49.172157    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9" event={"ID":"8c76eb36-5c29-41fa-8289-d8f08fba2181","Type":"ContainerDied","Data":"eccd86168122e976ae0bf9ee144d4b6bb0ce462315fafd25d506b52f7f8712af"}
Oct 30 19:55:49.182071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:49.172209    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9" event={"ID":"8c76eb36-5c29-41fa-8289-d8f08fba2181","Type":"ContainerDied","Data":"5483220e66417f21cfc33d47d3996b9c112af615cdf1da2ce3b5ea82d3cef922"}
Oct 30 19:55:49.182071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:49.172239    6030 scope.go:122] "RemoveContainer" containerID="eccd86168122e976ae0bf9ee144d4b6bb0ce462315fafd25d506b52f7f8712af"
Oct 30 19:55:49.182071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:49.172376    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"
Oct 30 19:55:49.246654 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:49.246621    6030 scope.go:122] "RemoveContainer" containerID="eccd86168122e976ae0bf9ee144d4b6bb0ce462315fafd25d506b52f7f8712af"
Oct 30 19:55:49.247657 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:55:49.247529    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eccd86168122e976ae0bf9ee144d4b6bb0ce462315fafd25d506b52f7f8712af\": container with ID starting with eccd86168122e976ae0bf9ee144d4b6bb0ce462315fafd25d506b52f7f8712af not found: ID does not exist" containerID="eccd86168122e976ae0bf9ee144d4b6bb0ce462315fafd25d506b52f7f8712af"
Oct 30 19:55:49.247657 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:49.247573    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eccd86168122e976ae0bf9ee144d4b6bb0ce462315fafd25d506b52f7f8712af"} err="failed to get container status \"eccd86168122e976ae0bf9ee144d4b6bb0ce462315fafd25d506b52f7f8712af\": rpc error: code = NotFound desc = could not find container \"eccd86168122e976ae0bf9ee144d4b6bb0ce462315fafd25d506b52f7f8712af\": container with ID starting with eccd86168122e976ae0bf9ee144d4b6bb0ce462315fafd25d506b52f7f8712af not found: ID does not exist"
Oct 30 19:55:49.253420 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:49.253377    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"]
Oct 30 19:55:49.268328 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:49.268303    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-hvgd9"]
Oct 30 19:55:51.216080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:51.215853    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c76eb36-5c29-41fa-8289-d8f08fba2181" path="/var/lib/kubelet/pods/8c76eb36-5c29-41fa-8289-d8f08fba2181/volumes"
Oct 30 19:55:51.640082 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:51.635248    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"]
Oct 30 19:55:51.640082 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:51.635968    6030 cpu_manager.go:414] "containerMap: removing container" podUID="d94e0001-0f96-4e02-bbba-fd81d82584e9" containerName="maas-api"
Oct 30 19:55:51.640082 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:51.635988    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d94e0001-0f96-4e02-bbba-fd81d82584e9" containerName="maas-api"
Oct 30 19:55:51.640082 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:51.636083    6030 cpu_manager.go:414] "containerMap: removing container" podUID="8c76eb36-5c29-41fa-8289-d8f08fba2181" containerName="istio-proxy"
Oct 30 19:55:51.640082 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:51.636093    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c76eb36-5c29-41fa-8289-d8f08fba2181" containerName="istio-proxy"
Oct 30 19:55:51.640082 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:51.636235    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c76eb36-5c29-41fa-8289-d8f08fba2181" containerName="istio-proxy"
Oct 30 19:55:51.640082 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:51.636252    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="d94e0001-0f96-4e02-bbba-fd81d82584e9" containerName="maas-api"
Oct 30 19:55:51.821292 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:51.821240    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"]
Oct 30 19:55:51.821722 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:51.821702    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:55:51.970445 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:51.970412    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-927bf8-serving-cert\""
Oct 30 19:55:51.997020 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:51.990252    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e3e551-15ee-4132-ad82-374dfc644e8d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd\" (UID: \"d5e3e551-15ee-4132-ad82-374dfc644e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:55:51.997020 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:51.990372    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e3e551-15ee-4132-ad82-374dfc644e8d-metrics-certs\") pod \"maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd\" (UID: \"d5e3e551-15ee-4132-ad82-374dfc644e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:55:51.997020 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:51.990411    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v47gd\" (UniqueName: \"kubernetes.io/projected/d5e3e551-15ee-4132-ad82-374dfc644e8d-kube-api-access-v47gd\") pod \"maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd\" (UID: \"d5e3e551-15ee-4132-ad82-374dfc644e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:55:51.997020 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:51.990450    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e3e551-15ee-4132-ad82-374dfc644e8d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd\" (UID: \"d5e3e551-15ee-4132-ad82-374dfc644e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:55:52.110113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:52.091547    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e3e551-15ee-4132-ad82-374dfc644e8d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd\" (UID: \"d5e3e551-15ee-4132-ad82-374dfc644e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:55:52.110113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:52.091665    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e3e551-15ee-4132-ad82-374dfc644e8d-metrics-certs\") pod \"maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd\" (UID: \"d5e3e551-15ee-4132-ad82-374dfc644e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:55:52.110113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:52.091699    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v47gd\" (UniqueName: \"kubernetes.io/projected/d5e3e551-15ee-4132-ad82-374dfc644e8d-kube-api-access-v47gd\") pod \"maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd\" (UID: \"d5e3e551-15ee-4132-ad82-374dfc644e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:55:52.110113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:52.091735    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e3e551-15ee-4132-ad82-374dfc644e8d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd\" (UID: \"d5e3e551-15ee-4132-ad82-374dfc644e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:55:52.110113 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:55:52.091928    6030 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-serving-cert: secret "maas-api-e2e-praxis-a-927bf8-serving-cert" not found
Oct 30 19:55:52.110113 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:55:52.091993    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5e3e551-15ee-4132-ad82-374dfc644e8d-maas-api-tls podName:d5e3e551-15ee-4132-ad82-374dfc644e8d nodeName:}" failed. No retries permitted until 2026-10-30 19:55:52.591971797 +0000 UTC m=+1431.507578650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5e3e551-15ee-4132-ad82-374dfc644e8d-maas-api-tls") pod "maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd" (UID: "d5e3e551-15ee-4132-ad82-374dfc644e8d") : secret "maas-api-e2e-praxis-a-927bf8-serving-cert" not found
Oct 30 19:55:52.110113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:52.092945    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e3e551-15ee-4132-ad82-374dfc644e8d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd\" (UID: \"d5e3e551-15ee-4132-ad82-374dfc644e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:55:52.110113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:52.096638    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e3e551-15ee-4132-ad82-374dfc644e8d-metrics-certs\") pod \"maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd\" (UID: \"d5e3e551-15ee-4132-ad82-374dfc644e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:55:52.153529 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:52.138127    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v47gd\" (UniqueName: \"kubernetes.io/projected/d5e3e551-15ee-4132-ad82-374dfc644e8d-kube-api-access-v47gd\") pod \"maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd\" (UID: \"d5e3e551-15ee-4132-ad82-374dfc644e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:55:52.622073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:52.615963    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e3e551-15ee-4132-ad82-374dfc644e8d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd\" (UID: \"d5e3e551-15ee-4132-ad82-374dfc644e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:55:52.642726 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:52.642077    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e3e551-15ee-4132-ad82-374dfc644e8d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd\" (UID: \"d5e3e551-15ee-4132-ad82-374dfc644e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:55:52.654070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:52.649759    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:55:52.946221 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:52.945174    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:52.961215 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:52.961162    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.201421 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:53.201313    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.289118 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:53.287568    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.318526 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:53.300323    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.344862 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:53.344019    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5" podUID="2db52234-3bbf-4a48-b905-dec230b1b3f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.403112 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:53.403008    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"]
Oct 30 19:55:53.510196 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:55:53.510104    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e3e551_15ee_4132_ad82_374dfc644e8d.slice/crio-7977af2df8b65aab9e78e5a0a77e429ddbb9451208de58e7f8ff6b5b3b485bda WatchSource:0}: Error finding container 7977af2df8b65aab9e78e5a0a77e429ddbb9451208de58e7f8ff6b5b3b485bda: Status 404 returned error can't find the container with id 7977af2df8b65aab9e78e5a0a77e429ddbb9451208de58e7f8ff6b5b3b485bda
Oct 30 19:55:53.626919 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:53.606139    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:54.504847 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:54.504808    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd" event={"ID":"d5e3e551-15ee-4132-ad82-374dfc644e8d","Type":"ContainerStarted","Data":"7977af2df8b65aab9e78e5a0a77e429ddbb9451208de58e7f8ff6b5b3b485bda"}
Oct 30 19:55:55.145569 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:55.145542    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"
Oct 30 19:55:55.512799 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:55.512704    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd" event={"ID":"d5e3e551-15ee-4132-ad82-374dfc644e8d","Type":"ContainerStarted","Data":"c98237544d73d2ba0d530ca7ca2c168807f164c6c019a3b665960d2e2dfceb43"}
Oct 30 19:55:55.512976 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:55.512847    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:55:55.538694 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:55:55.538643    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd" podStartSLOduration=3.637655088 podStartE2EDuration="4.538628666s" totalImagesPullingTime="900.973578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:55.537918327 +0000 UTC m=+1434.453525182" watchObservedRunningTime="2026-10-30 19:55:55.538628666 +0000 UTC m=+1434.454235521"
Oct 30 19:56:01.525013 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:01.524984    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:56:04.288964 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:56:04.288926    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:56:13.844376 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:13.844344    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"]
Oct 30 19:56:13.852095 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:13.852062    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:13.859913 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:13.859885    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-0eeba5-openshift-default-dockercfg-j457m\""
Oct 30 19:56:13.879403 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:13.879371    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"]
Oct 30 19:56:13.912794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:13.912236    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-data\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:13.912794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:13.912339    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-workload-socket\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:13.912794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:13.912409    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-credential-socket\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:13.912794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:13.912465    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-workload-certs\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:13.912794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:13.912492    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-token\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:13.912794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:13.912554    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj4b2\" (UniqueName: \"kubernetes.io/projected/c75bdfb0-89bd-4935-baae-55475fdb405a-kube-api-access-wj4b2\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:13.912794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:13.912587    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75bdfb0-89bd-4935-baae-55475fdb405a-istiod-ca-cert\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:13.912794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:13.912630    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-podinfo\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:13.912794 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:13.912708    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-envoy\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.013678 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.013643    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-workload-certs\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.013678 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.013682    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-token\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.013936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.013724    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj4b2\" (UniqueName: \"kubernetes.io/projected/c75bdfb0-89bd-4935-baae-55475fdb405a-kube-api-access-wj4b2\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.013936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.013746    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75bdfb0-89bd-4935-baae-55475fdb405a-istiod-ca-cert\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.013936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.013786    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-podinfo\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.013936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.013860    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-envoy\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.013936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.013909    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-data\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.014262 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.013990    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-workload-socket\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.014262 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.014079    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-credential-socket\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.014262 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.014163    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-workload-certs\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.014426 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.014390    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-data\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.014485 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.014434    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-workload-socket\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.014618 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.014590    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-credential-socket\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.015110 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.015085    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75bdfb0-89bd-4935-baae-55475fdb405a-istiod-ca-cert\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.016722 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.016695    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-envoy\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.016849 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.016746    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-podinfo\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.034563 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.034527    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj4b2\" (UniqueName: \"kubernetes.io/projected/c75bdfb0-89bd-4935-baae-55475fdb405a-kube-api-access-wj4b2\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.088072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.073079    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-token\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.170835 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.170802    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:14.713427 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.707282    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"]
Oct 30 19:56:14.733792 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.733166    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:56:14.733792 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.733279    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:56:14.733792 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:14.733324    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:56:15.674210 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:15.674172    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm" event={"ID":"c75bdfb0-89bd-4935-baae-55475fdb405a","Type":"ContainerStarted","Data":"80458fcd0987a05c5e049bbf172ce288143f933e723e9060d82b20e5789b24c2"}
Oct 30 19:56:15.674808 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:15.674771    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm" event={"ID":"c75bdfb0-89bd-4935-baae-55475fdb405a","Type":"ContainerStarted","Data":"a65ef055f13e1f2bb3af3aebfab1f83a38311ec69c79b27d6068581f9e8b097d"}
Oct 30 19:56:16.102282 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:16.102169    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:16.176242 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:16.176200    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:16.178277 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:16.178248    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 19:56:16.178521 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:16.178489    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm" podUID="c75bdfb0-89bd-4935-baae-55475fdb405a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 19:56:16.291465 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:16.291420    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:16.328186 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:16.328145    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:16.351956 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:16.351912    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5" podUID="2db52234-3bbf-4a48-b905-dec230b1b3f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:16.847067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:16.843188    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:16.899079 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:16.898014    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 19:56:16.926206 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:16.926140    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r" podUID="3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:16.930076 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:16.926520    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:56:17.105563 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.105462    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:17.108080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.106300    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:56:17.112150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.112117    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:17.153917 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.153853    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm" podStartSLOduration=4.153831693 podStartE2EDuration="4.153831693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:15.857933372 +0000 UTC m=+1454.773540229" watchObservedRunningTime="2026-10-30 19:56:17.153831693 +0000 UTC m=+1456.069438550"
Oct 30 19:56:17.185015 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.184983    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:17.300081 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.292526    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:17.300081 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.292885    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:17.302532 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.300616    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:56:17.332988 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.332950    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:17.333637 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.333613    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:56:17.589394 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.589357    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:56:17.590199 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.590164    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn" podUID="af2af060-ec91-4d0f-b61d-cfec82fc4e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:17.645859 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.644949    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:17.913524 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.902287    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:17.913524 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.904741    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:56:17.957497 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.954710    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:17.971373 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.959403    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:56:17.997106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:17.985225    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 19:56:18.058184 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:18.058151    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:56:18.098699 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:18.098658    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:18.110073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:18.110023    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:56:18.197987 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:18.197517    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:18.226337 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:18.226305    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"]
Oct 30 19:56:18.226824 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:18.226792    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6" podUID="f758324a-5fb7-4274-b85d-82767be18e82" containerName="maas-api" containerID="cri-o://a2dc1d26a67c30f3a2894d13f9aa3247c6a26b9c1edac18c5c0e61009a0a4660" gracePeriod=30
Oct 30 19:56:18.241303 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:18.241264    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:18.325391 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:18.325345    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:18.403895 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:18.403858    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:56:18.407378 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:18.405313    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:18.673189 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:18.673141    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:18.973614 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:18.972290    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r" podUID="3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:18.989267 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:18.987167    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 19:56:19.015144 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.006728    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 19:56:19.209749 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.209217    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:19.251223 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.244129    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:56:19.538254 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.538210    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 19:56:19.538552 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.538538    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:56:19.561857 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.561809    6030 generic.go:356] "Generic (PLEG): container finished" podID="f758324a-5fb7-4274-b85d-82767be18e82" containerID="a2dc1d26a67c30f3a2894d13f9aa3247c6a26b9c1edac18c5c0e61009a0a4660" exitCode=0
Oct 30 19:56:19.565937 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.565896    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6" event={"ID":"f758324a-5fb7-4274-b85d-82767be18e82","Type":"ContainerDied","Data":"a2dc1d26a67c30f3a2894d13f9aa3247c6a26b9c1edac18c5c0e61009a0a4660"}
Oct 30 19:56:19.579995 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.579335    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:56:19.632700 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.632591    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:56:19.804035 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.803943    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f758324a-5fb7-4274-b85d-82767be18e82-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f758324a-5fb7-4274-b85d-82767be18e82-maas-api-tls\") pod \"f758324a-5fb7-4274-b85d-82767be18e82\" (UID: \"f758324a-5fb7-4274-b85d-82767be18e82\") "
Oct 30 19:56:19.804035 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.804009    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f758324a-5fb7-4274-b85d-82767be18e82-kube-api-access-zddnw\" (UniqueName: \"kubernetes.io/projected/f758324a-5fb7-4274-b85d-82767be18e82-kube-api-access-zddnw\") pod \"f758324a-5fb7-4274-b85d-82767be18e82\" (UID: \"f758324a-5fb7-4274-b85d-82767be18e82\") "
Oct 30 19:56:19.804035 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.804055    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f758324a-5fb7-4274-b85d-82767be18e82-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f758324a-5fb7-4274-b85d-82767be18e82-metrics-certs\") pod \"f758324a-5fb7-4274-b85d-82767be18e82\" (UID: \"f758324a-5fb7-4274-b85d-82767be18e82\") "
Oct 30 19:56:19.804378 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.804142    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f758324a-5fb7-4274-b85d-82767be18e82-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f758324a-5fb7-4274-b85d-82767be18e82-openshift-service-ca\") pod \"f758324a-5fb7-4274-b85d-82767be18e82\" (UID: \"f758324a-5fb7-4274-b85d-82767be18e82\") "
Oct 30 19:56:19.815721 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.804853    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f758324a-5fb7-4274-b85d-82767be18e82-openshift-service-ca" pod "f758324a-5fb7-4274-b85d-82767be18e82" (UID: "f758324a-5fb7-4274-b85d-82767be18e82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:19.852676 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.852619    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f758324a-5fb7-4274-b85d-82767be18e82-kube-api-access-zddnw" pod "f758324a-5fb7-4274-b85d-82767be18e82" (UID: "f758324a-5fb7-4274-b85d-82767be18e82"). InnerVolumeSpecName "kube-api-access-zddnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:19.853696 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.853650    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f758324a-5fb7-4274-b85d-82767be18e82-maas-api-tls" pod "f758324a-5fb7-4274-b85d-82767be18e82" (UID: "f758324a-5fb7-4274-b85d-82767be18e82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:19.853992 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.853961    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f758324a-5fb7-4274-b85d-82767be18e82-metrics-certs" pod "f758324a-5fb7-4274-b85d-82767be18e82" (UID: "f758324a-5fb7-4274-b85d-82767be18e82"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:19.924098 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.908103    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f758324a-5fb7-4274-b85d-82767be18e82-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:19.924098 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.908143    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f758324a-5fb7-4274-b85d-82767be18e82-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:19.924098 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.908158    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zddnw\" (UniqueName: \"kubernetes.io/projected/f758324a-5fb7-4274-b85d-82767be18e82-kube-api-access-zddnw\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:19.924098 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:19.908173    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f758324a-5fb7-4274-b85d-82767be18e82-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:20.233256 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.232810    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"]
Oct 30 19:56:20.233632 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.233570    6030 cpu_manager.go:414] "containerMap: removing container" podUID="f758324a-5fb7-4274-b85d-82767be18e82" containerName="maas-api"
Oct 30 19:56:20.233632 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.233585    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f758324a-5fb7-4274-b85d-82767be18e82" containerName="maas-api"
Oct 30 19:56:20.233703 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.233691    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="f758324a-5fb7-4274-b85d-82767be18e82" containerName="maas-api"
Oct 30 19:56:20.238560 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.237156    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"
Oct 30 19:56:20.245815 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.245765    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"]
Oct 30 19:56:20.248092 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.248014    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-b24a1a\""
Oct 30 19:56:20.248345 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.248319    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-b24a1a-dockercfg-fcnx7\""
Oct 30 19:56:20.279077 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.278830    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"]
Oct 30 19:56:20.290920 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.283337    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"
Oct 30 19:56:20.318295 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.317663    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"]
Oct 30 19:56:20.324949 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.324915    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv\" (UID: \"bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"
Oct 30 19:56:20.325304 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.325278    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frqb7\" (UniqueName: \"kubernetes.io/projected/20147f52-0076-4eb0-aad7-5f262111a38b-kube-api-access-frqb7\") pod \"payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs\" (UID: \"20147f52-0076-4eb0-aad7-5f262111a38b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"
Oct 30 19:56:20.325604 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.325583    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrtfb\" (UniqueName: \"kubernetes.io/projected/bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35-kube-api-access-zrtfb\") pod \"payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv\" (UID: \"bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"
Oct 30 19:56:20.325913 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.325792    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20147f52-0076-4eb0-aad7-5f262111a38b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs\" (UID: \"20147f52-0076-4eb0-aad7-5f262111a38b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"
Oct 30 19:56:20.433847 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.433791    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv\" (UID: \"bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"
Oct 30 19:56:20.435178 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.435146    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frqb7\" (UniqueName: \"kubernetes.io/projected/20147f52-0076-4eb0-aad7-5f262111a38b-kube-api-access-frqb7\") pod \"payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs\" (UID: \"20147f52-0076-4eb0-aad7-5f262111a38b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"
Oct 30 19:56:20.435834 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.435810    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrtfb\" (UniqueName: \"kubernetes.io/projected/bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35-kube-api-access-zrtfb\") pod \"payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv\" (UID: \"bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"
Oct 30 19:56:20.436304 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.436281    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20147f52-0076-4eb0-aad7-5f262111a38b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs\" (UID: \"20147f52-0076-4eb0-aad7-5f262111a38b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"
Oct 30 19:56:20.437418 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.434960    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv\" (UID: \"bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"
Oct 30 19:56:20.437586 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.437238    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20147f52-0076-4eb0-aad7-5f262111a38b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs\" (UID: \"20147f52-0076-4eb0-aad7-5f262111a38b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"
Oct 30 19:56:20.508891 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.508805    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrtfb\" (UniqueName: \"kubernetes.io/projected/bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35-kube-api-access-zrtfb\") pod \"payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv\" (UID: \"bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"
Oct 30 19:56:20.510370 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.510289    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frqb7\" (UniqueName: \"kubernetes.io/projected/20147f52-0076-4eb0-aad7-5f262111a38b-kube-api-access-frqb7\") pod \"payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs\" (UID: \"20147f52-0076-4eb0-aad7-5f262111a38b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"
Oct 30 19:56:20.609350 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.609314    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"
Oct 30 19:56:20.631238 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.631208    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6" event={"ID":"f758324a-5fb7-4274-b85d-82767be18e82","Type":"ContainerDied","Data":"bd82b1fa5f855e16f05a5fd1cc094f7e6f46bdd5b221bfce6b39b08bb055c7b4"}
Oct 30 19:56:20.631376 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.631250    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"
Oct 30 19:56:20.631376 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.631301    6030 scope.go:122] "RemoveContainer" containerID="a2dc1d26a67c30f3a2894d13f9aa3247c6a26b9c1edac18c5c0e61009a0a4660"
Oct 30 19:56:20.639359 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.639339    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"
Oct 30 19:56:20.667717 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.667690    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"]
Oct 30 19:56:20.712639 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.712338    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f03120-6f9b89b48d-n2kx6"]
Oct 30 19:56:20.992386 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.992321    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"]
Oct 30 19:56:21.000162 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:20.999014    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"]
Oct 30 19:56:21.101990 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.101954    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"]
Oct 30 19:56:21.108566 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.108535    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"
Oct 30 19:56:21.115091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.113291    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"]
Oct 30 19:56:21.115091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.114229    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-0eeba5-dockercfg-smhxf\""
Oct 30 19:56:21.115091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.114682    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-0eeba5\""
Oct 30 19:56:21.259071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.258181    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgj8v\" (UniqueName: \"kubernetes.io/projected/aea211f6-4038-4d8f-a09f-581a68b6c73d-kube-api-access-wgj8v\") pod \"payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz\" (UID: \"aea211f6-4038-4d8f-a09f-581a68b6c73d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"
Oct 30 19:56:21.259071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.258384    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea211f6-4038-4d8f-a09f-581a68b6c73d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz\" (UID: \"aea211f6-4038-4d8f-a09f-581a68b6c73d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"
Oct 30 19:56:21.349149 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.343449    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f758324a-5fb7-4274-b85d-82767be18e82" path="/var/lib/kubelet/pods/f758324a-5fb7-4274-b85d-82767be18e82/volumes"
Oct 30 19:56:21.363849 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.361816    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea211f6-4038-4d8f-a09f-581a68b6c73d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz\" (UID: \"aea211f6-4038-4d8f-a09f-581a68b6c73d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"
Oct 30 19:56:21.363849 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.361975    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgj8v\" (UniqueName: \"kubernetes.io/projected/aea211f6-4038-4d8f-a09f-581a68b6c73d-kube-api-access-wgj8v\") pod \"payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz\" (UID: \"aea211f6-4038-4d8f-a09f-581a68b6c73d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"
Oct 30 19:56:21.363849 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.363390    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea211f6-4038-4d8f-a09f-581a68b6c73d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz\" (UID: \"aea211f6-4038-4d8f-a09f-581a68b6c73d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"
Oct 30 19:56:21.411125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.404307    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgj8v\" (UniqueName: \"kubernetes.io/projected/aea211f6-4038-4d8f-a09f-581a68b6c73d-kube-api-access-wgj8v\") pod \"payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz\" (UID: \"aea211f6-4038-4d8f-a09f-581a68b6c73d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"
Oct 30 19:56:21.454070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.449815    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"
Oct 30 19:56:21.639815 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.639784    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv" event={"ID":"bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35","Type":"ContainerStarted","Data":"7520eb36410c2ce41608bc2e4b0265e0958af5dfbec9bdc40a01097b1c76f0f1"}
Oct 30 19:56:21.640034 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.639833    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv" event={"ID":"bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35","Type":"ContainerStarted","Data":"6ba740b5f831ea7b53bff5c2f489ff7643d4af1ab045b8f379ddf2d93aa1f27b"}
Oct 30 19:56:21.640034 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.639859    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"
Oct 30 19:56:21.642701 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.642674    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs" event={"ID":"20147f52-0076-4eb0-aad7-5f262111a38b","Type":"ContainerStarted","Data":"e0c09b9c4034a92b473c1cc58e3c81d3352ac5ae3e5d3345b0a38670161170f5"}
Oct 30 19:56:21.642819 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.642721    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs" event={"ID":"20147f52-0076-4eb0-aad7-5f262111a38b","Type":"ContainerStarted","Data":"3665bc50b01ef1c7c1c20c29e53a02af147bb73262aed31aa6992d4868b3d304"}
Oct 30 19:56:21.642819 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.642750    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"
Oct 30 19:56:21.664623 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.664573    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv" podStartSLOduration=1.664559311 podStartE2EDuration="1.664559311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:21.662190147 +0000 UTC m=+1460.577797004" watchObservedRunningTime="2026-10-30 19:56:21.664559311 +0000 UTC m=+1460.580166191"
Oct 30 19:56:21.703716 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.703656    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs" podStartSLOduration=1.703637234 podStartE2EDuration="1.703637234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:21.702933785 +0000 UTC m=+1460.618540641" watchObservedRunningTime="2026-10-30 19:56:21.703637234 +0000 UTC m=+1460.619244094"
Oct 30 19:56:21.766673 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:21.766629    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"]
Oct 30 19:56:21.769187 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:56:21.769150    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea211f6_4038_4d8f_a09f_581a68b6c73d.slice/crio-7a0acb08ac9944f20cc703e6a9a1dcf50f5cd687bf8e1f3dea4d9d4a9f45ac71 WatchSource:0}: Error finding container 7a0acb08ac9944f20cc703e6a9a1dcf50f5cd687bf8e1f3dea4d9d4a9f45ac71: Status 404 returned error can't find the container with id 7a0acb08ac9944f20cc703e6a9a1dcf50f5cd687bf8e1f3dea4d9d4a9f45ac71
Oct 30 19:56:22.650256 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:22.650214    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz" event={"ID":"aea211f6-4038-4d8f-a09f-581a68b6c73d","Type":"ContainerStarted","Data":"6836db3b82359fac828050257f1f58f7df30fc63fec5d50c8f1d7c46383c25d4"}
Oct 30 19:56:22.650651 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:22.650303    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz" event={"ID":"aea211f6-4038-4d8f-a09f-581a68b6c73d","Type":"ContainerStarted","Data":"7a0acb08ac9944f20cc703e6a9a1dcf50f5cd687bf8e1f3dea4d9d4a9f45ac71"}
Oct 30 19:56:22.650651 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:22.650497    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"
Oct 30 19:56:26.098228 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:26.098194    6030 patch_prober.go:29] interesting pod/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:56:26.098764 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:26.098265    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz" podUID="a29ff919-34f3-412f-adf8-d331698289e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:28.704801 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:28.704767    6030 generic.go:356] "Generic (PLEG): container finished" podID="a29ff919-34f3-412f-adf8-d331698289e9" containerID="0c711057cba171197487995af0ab346ee6fbe181ca195025e5cece206601db4a" exitCode=0
Oct 30 19:56:28.705278 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:28.704850    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz" event={"ID":"a29ff919-34f3-412f-adf8-d331698289e9","Type":"ContainerDied","Data":"0c711057cba171197487995af0ab346ee6fbe181ca195025e5cece206601db4a"}
Oct 30 19:56:28.705328 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:28.705307    6030 scope.go:122] "RemoveContainer" containerID="0c711057cba171197487995af0ab346ee6fbe181ca195025e5cece206601db4a"
Oct 30 19:56:28.706902 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:28.706868    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:56:28.707007 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:28.706954    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:56:28.707104 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:28.707006    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:56:28.742561 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:28.742516    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz" podStartSLOduration=7.742502891 podStartE2EDuration="7.742502891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:22.69621225 +0000 UTC m=+1461.611819107" watchObservedRunningTime="2026-10-30 19:56:28.742502891 +0000 UTC m=+1467.658109746"
Oct 30 19:56:29.737070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:29.731269    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz" event={"ID":"a29ff919-34f3-412f-adf8-d331698289e9","Type":"ContainerStarted","Data":"0494cffa8329d3f0cbf5d86a1393fad15a3105dcf35f943ef00aed5a79aa0a50"}
Oct 30 19:56:30.137279 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.134615    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:56:30.137279 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.135022    6030 patch_prober.go:29] interesting pod/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:56:30.137279 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.135099    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz" podUID="a29ff919-34f3-412f-adf8-d331698289e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 19:56:30.137279 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.137239    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"]
Oct 30 19:56:30.150456 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.141314    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 19:56:30.154315 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.154286    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-0eeba5-serving-cert\""
Oct 30 19:56:30.163917 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.163829    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"]
Oct 30 19:56:30.234182 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.234150    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8790b73a-43cc-496c-ab9e-250aabb2306f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw\" (UID: \"8790b73a-43cc-496c-ab9e-250aabb2306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 19:56:30.234366 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.234285    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8790b73a-43cc-496c-ab9e-250aabb2306f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw\" (UID: \"8790b73a-43cc-496c-ab9e-250aabb2306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 19:56:30.234366 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.234330    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8790b73a-43cc-496c-ab9e-250aabb2306f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw\" (UID: \"8790b73a-43cc-496c-ab9e-250aabb2306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 19:56:30.234445 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.234419    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sxhf\" (UniqueName: \"kubernetes.io/projected/8790b73a-43cc-496c-ab9e-250aabb2306f-kube-api-access-7sxhf\") pod \"maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw\" (UID: \"8790b73a-43cc-496c-ab9e-250aabb2306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 19:56:30.335511 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.335476    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sxhf\" (UniqueName: \"kubernetes.io/projected/8790b73a-43cc-496c-ab9e-250aabb2306f-kube-api-access-7sxhf\") pod \"maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw\" (UID: \"8790b73a-43cc-496c-ab9e-250aabb2306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 19:56:30.335727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.335622    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8790b73a-43cc-496c-ab9e-250aabb2306f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw\" (UID: \"8790b73a-43cc-496c-ab9e-250aabb2306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 19:56:30.335804 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.335725    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8790b73a-43cc-496c-ab9e-250aabb2306f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw\" (UID: \"8790b73a-43cc-496c-ab9e-250aabb2306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 19:56:30.335804 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.335779    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8790b73a-43cc-496c-ab9e-250aabb2306f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw\" (UID: \"8790b73a-43cc-496c-ab9e-250aabb2306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 19:56:30.336552 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.336503    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8790b73a-43cc-496c-ab9e-250aabb2306f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw\" (UID: \"8790b73a-43cc-496c-ab9e-250aabb2306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 19:56:30.338361 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.338338    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8790b73a-43cc-496c-ab9e-250aabb2306f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw\" (UID: \"8790b73a-43cc-496c-ab9e-250aabb2306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 19:56:30.338486 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.338367    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8790b73a-43cc-496c-ab9e-250aabb2306f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw\" (UID: \"8790b73a-43cc-496c-ab9e-250aabb2306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 19:56:30.349658 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.349634    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sxhf\" (UniqueName: \"kubernetes.io/projected/8790b73a-43cc-496c-ab9e-250aabb2306f-kube-api-access-7sxhf\") pod \"maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw\" (UID: \"8790b73a-43cc-496c-ab9e-250aabb2306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 19:56:30.465121 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.464288    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 19:56:30.807746 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:30.807204    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"]
Oct 30 19:56:31.135811 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:31.135768    6030 patch_prober.go:29] interesting pod/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:56:31.135986 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:31.135836    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz" podUID="a29ff919-34f3-412f-adf8-d331698289e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 19:56:31.822395 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:31.822295    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw" event={"ID":"8790b73a-43cc-496c-ab9e-250aabb2306f","Type":"ContainerStarted","Data":"856abb2d76bf642b2bb6e582652bfb2295068e91e7c1d1aa95ab46be5fdda1da"}
Oct 30 19:56:31.822395 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:31.822357    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw" event={"ID":"8790b73a-43cc-496c-ab9e-250aabb2306f","Type":"ContainerStarted","Data":"b8b0f68d7e5e902ffea081eb9964ffe00c7344cb0308f8f2f90f37192c29a122"}
Oct 30 19:56:31.822395 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:31.822389    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 19:56:32.135260 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:32.135223    6030 patch_prober.go:29] interesting pod/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:56:32.135442 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:32.135282    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz" podUID="a29ff919-34f3-412f-adf8-d331698289e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 19:56:32.653720 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:32.653689    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"
Oct 30 19:56:32.654088 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:32.653984    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"
Oct 30 19:56:32.692070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:32.689427    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw" podStartSLOduration=2.186640432 podStartE2EDuration="2.689404122s" totalImagesPullingTime="502.76369ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:31.848300363 +0000 UTC m=+1470.763907220" watchObservedRunningTime="2026-10-30 19:56:32.689404122 +0000 UTC m=+1471.605010978"
Oct 30 19:56:33.135350 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:33.135318    6030 patch_prober.go:29] interesting pod/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:56:33.135750 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:33.135374    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz" podUID="a29ff919-34f3-412f-adf8-d331698289e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 19:56:33.659655 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:33.659623    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"
Oct 30 19:56:34.150081 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:34.139803    6030 patch_prober.go:29] interesting pod/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:56:34.150081 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:34.139864    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz" podUID="a29ff919-34f3-412f-adf8-d331698289e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 19:56:34.311844 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:56:34.311812    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:56:35.205409 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:35.205369    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:56:35.206295 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:35.206258    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:56:35.207549 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:35.207526    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:56:37.834419 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:37.834389    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 19:56:39.933176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:39.933130    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:39.934132 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:39.934099    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:56:39.935734 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:39.935706    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r" podUID="3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:40.113282 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:40.113237    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 19:56:40.120072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:40.117139    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:40.213250 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:40.213149    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 19:56:40.296085 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:40.296021    6030 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:40.336178 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:40.336135    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5" podUID="2db52234-3bbf-4a48-b905-dec230b1b3f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 19:56:40.596504 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:40.596172    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:40.936227 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:40.936178    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r" podUID="3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:40.937205 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:40.936796    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:40.938667 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:40.937813    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:56:40.952262 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:40.952239    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:56:41.063835 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:41.062919    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 19:56:41.090761 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:41.089987    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:41.127355 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:41.126298    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:41.127355 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:41.127111    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:56:41.157103 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:41.149872    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 19:56:41.420632 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:41.420602    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:56:49.560648 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:49.560611    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"]
Oct 30 19:56:50.896293 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:50.896253    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"]
Oct 30 19:56:50.900403 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:50.900382    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"
Oct 30 19:56:50.912286 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:50.912241    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-7eaeed\""
Oct 30 19:56:50.912423 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:50.912362    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-7eaeed-dockercfg-tnf5t\""
Oct 30 19:56:50.917076 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:50.917022    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"]
Oct 30 19:56:50.928652 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:50.928602    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"]
Oct 30 19:56:50.933682 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:50.933656    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"
Oct 30 19:56:50.953710 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:50.953685    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"]
Oct 30 19:56:50.985301 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:50.977977    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d043a6ac-9ac9-4f66-9669-e253eea8b381-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668\" (UID: \"d043a6ac-9ac9-4f66-9669-e253eea8b381\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"
Oct 30 19:56:50.985301 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:50.978124    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4290bb83-6138-4d07-a193-f8fb4f5b2fb4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p\" (UID: \"4290bb83-6138-4d07-a193-f8fb4f5b2fb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"
Oct 30 19:56:50.985301 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:50.978178    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t5v4\" (UniqueName: \"kubernetes.io/projected/d043a6ac-9ac9-4f66-9669-e253eea8b381-kube-api-access-4t5v4\") pod \"payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668\" (UID: \"d043a6ac-9ac9-4f66-9669-e253eea8b381\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"
Oct 30 19:56:50.985301 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:50.978222    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rlj\" (UniqueName: \"kubernetes.io/projected/4290bb83-6138-4d07-a193-f8fb4f5b2fb4-kube-api-access-g5rlj\") pod \"payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p\" (UID: \"4290bb83-6138-4d07-a193-f8fb4f5b2fb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"
Oct 30 19:56:51.094106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:51.091684    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d043a6ac-9ac9-4f66-9669-e253eea8b381-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668\" (UID: \"d043a6ac-9ac9-4f66-9669-e253eea8b381\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"
Oct 30 19:56:51.094106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:51.091813    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4290bb83-6138-4d07-a193-f8fb4f5b2fb4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p\" (UID: \"4290bb83-6138-4d07-a193-f8fb4f5b2fb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"
Oct 30 19:56:51.094106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:51.091862    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t5v4\" (UniqueName: \"kubernetes.io/projected/d043a6ac-9ac9-4f66-9669-e253eea8b381-kube-api-access-4t5v4\") pod \"payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668\" (UID: \"d043a6ac-9ac9-4f66-9669-e253eea8b381\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"
Oct 30 19:56:51.094106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:51.091906    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rlj\" (UniqueName: \"kubernetes.io/projected/4290bb83-6138-4d07-a193-f8fb4f5b2fb4-kube-api-access-g5rlj\") pod \"payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p\" (UID: \"4290bb83-6138-4d07-a193-f8fb4f5b2fb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"
Oct 30 19:56:51.094106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:51.093169    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d043a6ac-9ac9-4f66-9669-e253eea8b381-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668\" (UID: \"d043a6ac-9ac9-4f66-9669-e253eea8b381\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"
Oct 30 19:56:51.094106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:51.093867    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4290bb83-6138-4d07-a193-f8fb4f5b2fb4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p\" (UID: \"4290bb83-6138-4d07-a193-f8fb4f5b2fb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"
Oct 30 19:56:51.226202 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:51.225949    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rlj\" (UniqueName: \"kubernetes.io/projected/4290bb83-6138-4d07-a193-f8fb4f5b2fb4-kube-api-access-g5rlj\") pod \"payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p\" (UID: \"4290bb83-6138-4d07-a193-f8fb4f5b2fb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"
Oct 30 19:56:51.238856 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:51.236730    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t5v4\" (UniqueName: \"kubernetes.io/projected/d043a6ac-9ac9-4f66-9669-e253eea8b381-kube-api-access-4t5v4\") pod \"payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668\" (UID: \"d043a6ac-9ac9-4f66-9669-e253eea8b381\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"
Oct 30 19:56:51.252826 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:51.252798    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"
Oct 30 19:56:51.523132 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:51.522243    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"
Oct 30 19:56:51.617381 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:51.617352    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"]
Oct 30 19:56:51.618594 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:56:51.618570    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4290bb83_6138_4d07_a193_f8fb4f5b2fb4.slice/crio-6c66c34d5e61bc16da172846008f9fc44172a5c57a5787ec29843fedfcde3eba WatchSource:0}: Error finding container 6c66c34d5e61bc16da172846008f9fc44172a5c57a5787ec29843fedfcde3eba: Status 404 returned error can't find the container with id 6c66c34d5e61bc16da172846008f9fc44172a5c57a5787ec29843fedfcde3eba
Oct 30 19:56:51.883760 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:51.883737    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"]
Oct 30 19:56:51.886785 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:56:51.886748    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd043a6ac_9ac9_4f66_9669_e253eea8b381.slice/crio-0b58e50e74a9c7312f75ff8d39b6c50d641382f45addd56c404c7c726ff708d3 WatchSource:0}: Error finding container 0b58e50e74a9c7312f75ff8d39b6c50d641382f45addd56c404c7c726ff708d3: Status 404 returned error can't find the container with id 0b58e50e74a9c7312f75ff8d39b6c50d641382f45addd56c404c7c726ff708d3
Oct 30 19:56:52.188675 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:52.188635    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668" event={"ID":"d043a6ac-9ac9-4f66-9669-e253eea8b381","Type":"ContainerStarted","Data":"cce1190d212eb6fc003eb3908ae34dc191bba302c35a0f2790b0be6ff491af7b"}
Oct 30 19:56:52.189160 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:52.188736    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668" event={"ID":"d043a6ac-9ac9-4f66-9669-e253eea8b381","Type":"ContainerStarted","Data":"0b58e50e74a9c7312f75ff8d39b6c50d641382f45addd56c404c7c726ff708d3"}
Oct 30 19:56:52.189160 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:52.188767    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"
Oct 30 19:56:52.190302 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:52.190270    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p" event={"ID":"4290bb83-6138-4d07-a193-f8fb4f5b2fb4","Type":"ContainerStarted","Data":"964287b30e0cc22022fa2623262f532c4906acaa2527db2eae626b21a314e550"}
Oct 30 19:56:52.190457 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:52.190331    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"
Oct 30 19:56:52.190457 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:52.190354    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p" event={"ID":"4290bb83-6138-4d07-a193-f8fb4f5b2fb4","Type":"ContainerStarted","Data":"6c66c34d5e61bc16da172846008f9fc44172a5c57a5787ec29843fedfcde3eba"}
Oct 30 19:56:52.213203 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:52.213155    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668" podStartSLOduration=2.213141427 podStartE2EDuration="2.213141427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:52.21031066 +0000 UTC m=+1491.125917518" watchObservedRunningTime="2026-10-30 19:56:52.213141427 +0000 UTC m=+1491.128748304"
Oct 30 19:56:52.242292 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:52.242228    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p" podStartSLOduration=2.24220991 podStartE2EDuration="2.24220991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:52.239526549 +0000 UTC m=+1491.155133406" watchObservedRunningTime="2026-10-30 19:56:52.24220991 +0000 UTC m=+1491.157816767"
Oct 30 19:56:53.453370 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:53.453333    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"]
Oct 30 19:56:53.453858 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:53.453530    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd" podUID="d5e3e551-15ee-4132-ad82-374dfc644e8d" containerName="maas-api" containerID="cri-o://c98237544d73d2ba0d530ca7ca2c168807f164c6c019a3b665960d2e2dfceb43" gracePeriod=30
Oct 30 19:56:53.940090 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:53.940063    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:56:54.014844 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.014809    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e3e551-15ee-4132-ad82-374dfc644e8d-kube-api-access-v47gd\" (UniqueName: \"kubernetes.io/projected/d5e3e551-15ee-4132-ad82-374dfc644e8d-kube-api-access-v47gd\") pod \"d5e3e551-15ee-4132-ad82-374dfc644e8d\" (UID: \"d5e3e551-15ee-4132-ad82-374dfc644e8d\") "
Oct 30 19:56:54.015102 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.014898    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e3e551-15ee-4132-ad82-374dfc644e8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e3e551-15ee-4132-ad82-374dfc644e8d-maas-api-tls\") pod \"d5e3e551-15ee-4132-ad82-374dfc644e8d\" (UID: \"d5e3e551-15ee-4132-ad82-374dfc644e8d\") "
Oct 30 19:56:54.015102 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.015088    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e3e551-15ee-4132-ad82-374dfc644e8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e3e551-15ee-4132-ad82-374dfc644e8d-metrics-certs\") pod \"d5e3e551-15ee-4132-ad82-374dfc644e8d\" (UID: \"d5e3e551-15ee-4132-ad82-374dfc644e8d\") "
Oct 30 19:56:54.015224 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.015141    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e3e551-15ee-4132-ad82-374dfc644e8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e3e551-15ee-4132-ad82-374dfc644e8d-openshift-service-ca\") pod \"d5e3e551-15ee-4132-ad82-374dfc644e8d\" (UID: \"d5e3e551-15ee-4132-ad82-374dfc644e8d\") "
Oct 30 19:56:54.015698 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.015658    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e3e551-15ee-4132-ad82-374dfc644e8d-openshift-service-ca" pod "d5e3e551-15ee-4132-ad82-374dfc644e8d" (UID: "d5e3e551-15ee-4132-ad82-374dfc644e8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:54.019963 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.019937    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e3e551-15ee-4132-ad82-374dfc644e8d-kube-api-access-v47gd" pod "d5e3e551-15ee-4132-ad82-374dfc644e8d" (UID: "d5e3e551-15ee-4132-ad82-374dfc644e8d"). InnerVolumeSpecName "kube-api-access-v47gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:54.020110 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.020084    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e3e551-15ee-4132-ad82-374dfc644e8d-metrics-certs" pod "d5e3e551-15ee-4132-ad82-374dfc644e8d" (UID: "d5e3e551-15ee-4132-ad82-374dfc644e8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:54.021375 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.021349    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e3e551-15ee-4132-ad82-374dfc644e8d-maas-api-tls" pod "d5e3e551-15ee-4132-ad82-374dfc644e8d" (UID: "d5e3e551-15ee-4132-ad82-374dfc644e8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:54.116459 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.116429    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e3e551-15ee-4132-ad82-374dfc644e8d-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:54.116459 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.116457    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e3e551-15ee-4132-ad82-374dfc644e8d-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:54.116656 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.116468    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v47gd\" (UniqueName: \"kubernetes.io/projected/d5e3e551-15ee-4132-ad82-374dfc644e8d-kube-api-access-v47gd\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:54.116656 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.116478    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e3e551-15ee-4132-ad82-374dfc644e8d-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:54.209122 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.208880    6030 generic.go:356] "Generic (PLEG): container finished" podID="d5e3e551-15ee-4132-ad82-374dfc644e8d" containerID="c98237544d73d2ba0d530ca7ca2c168807f164c6c019a3b665960d2e2dfceb43" exitCode=0
Oct 30 19:56:54.209122 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.209103    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd" event={"ID":"d5e3e551-15ee-4132-ad82-374dfc644e8d","Type":"ContainerDied","Data":"c98237544d73d2ba0d530ca7ca2c168807f164c6c019a3b665960d2e2dfceb43"}
Oct 30 19:56:54.209364 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.209156    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd" event={"ID":"d5e3e551-15ee-4132-ad82-374dfc644e8d","Type":"ContainerDied","Data":"7977af2df8b65aab9e78e5a0a77e429ddbb9451208de58e7f8ff6b5b3b485bda"}
Oct 30 19:56:54.209364 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.209186    6030 scope.go:122] "RemoveContainer" containerID="c98237544d73d2ba0d530ca7ca2c168807f164c6c019a3b665960d2e2dfceb43"
Oct 30 19:56:54.209364 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.209350    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"
Oct 30 19:56:54.229754 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.229730    6030 scope.go:122] "RemoveContainer" containerID="c98237544d73d2ba0d530ca7ca2c168807f164c6c019a3b665960d2e2dfceb43"
Oct 30 19:56:54.230169 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:56:54.230136    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98237544d73d2ba0d530ca7ca2c168807f164c6c019a3b665960d2e2dfceb43\": container with ID starting with c98237544d73d2ba0d530ca7ca2c168807f164c6c019a3b665960d2e2dfceb43 not found: ID does not exist" containerID="c98237544d73d2ba0d530ca7ca2c168807f164c6c019a3b665960d2e2dfceb43"
Oct 30 19:56:54.230295 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.230179    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98237544d73d2ba0d530ca7ca2c168807f164c6c019a3b665960d2e2dfceb43"} err="failed to get container status \"c98237544d73d2ba0d530ca7ca2c168807f164c6c019a3b665960d2e2dfceb43\": rpc error: code = NotFound desc = could not find container \"c98237544d73d2ba0d530ca7ca2c168807f164c6c019a3b665960d2e2dfceb43\": container with ID starting with c98237544d73d2ba0d530ca7ca2c168807f164c6c019a3b665960d2e2dfceb43 not found: ID does not exist"
Oct 30 19:56:54.247125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.247082    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"]
Oct 30 19:56:54.250612 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.250590    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-927bf8-679d946dc9-r6xfd"]
Oct 30 19:56:54.561714 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.561623    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx" podUID="755da169-322e-451f-b4ac-90053aa2b956" containerName="payload-pre-processing" containerID="cri-o://5e8f0f8be5f0ef3befa5fd647e1cbb841fd1dbee7bad264ae65ebf6813325860" gracePeriod=25
Oct 30 19:56:54.828785 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.828721    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"
Oct 30 19:56:54.927739 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.927698    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/755da169-322e-451f-b4ac-90053aa2b956-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/755da169-322e-451f-b4ac-90053aa2b956-plugins-config-volume\") pod \"755da169-322e-451f-b4ac-90053aa2b956\" (UID: \"755da169-322e-451f-b4ac-90053aa2b956\") "
Oct 30 19:56:54.927939 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.927785    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/755da169-322e-451f-b4ac-90053aa2b956-kube-api-access-tffdt\" (UniqueName: \"kubernetes.io/projected/755da169-322e-451f-b4ac-90053aa2b956-kube-api-access-tffdt\") pod \"755da169-322e-451f-b4ac-90053aa2b956\" (UID: \"755da169-322e-451f-b4ac-90053aa2b956\") "
Oct 30 19:56:54.928090 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.928038    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/755da169-322e-451f-b4ac-90053aa2b956-plugins-config-volume" pod "755da169-322e-451f-b4ac-90053aa2b956" (UID: "755da169-322e-451f-b4ac-90053aa2b956"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:54.928331 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.928317    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/755da169-322e-451f-b4ac-90053aa2b956-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:54.930113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:54.930079    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755da169-322e-451f-b4ac-90053aa2b956-kube-api-access-tffdt" pod "755da169-322e-451f-b4ac-90053aa2b956" (UID: "755da169-322e-451f-b4ac-90053aa2b956"). InnerVolumeSpecName "kube-api-access-tffdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:55.030028 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.029993    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tffdt\" (UniqueName: \"kubernetes.io/projected/755da169-322e-451f-b4ac-90053aa2b956-kube-api-access-tffdt\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:55.160644 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.160608    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"]
Oct 30 19:56:55.161388 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.161367    6030 cpu_manager.go:414] "containerMap: removing container" podUID="755da169-322e-451f-b4ac-90053aa2b956" containerName="payload-pre-processing"
Oct 30 19:56:55.161388 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.161385    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="755da169-322e-451f-b4ac-90053aa2b956" containerName="payload-pre-processing"
Oct 30 19:56:55.161532 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.161418    6030 cpu_manager.go:414] "containerMap: removing container" podUID="d5e3e551-15ee-4132-ad82-374dfc644e8d" containerName="maas-api"
Oct 30 19:56:55.161532 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.161425    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e3e551-15ee-4132-ad82-374dfc644e8d" containerName="maas-api"
Oct 30 19:56:55.161532 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.161526    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e3e551-15ee-4132-ad82-374dfc644e8d" containerName="maas-api"
Oct 30 19:56:55.161629 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.161537    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="755da169-322e-451f-b4ac-90053aa2b956" containerName="payload-pre-processing"
Oct 30 19:56:55.165006 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.164990    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"
Oct 30 19:56:55.171622 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.171264    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-54d4c4\""
Oct 30 19:56:55.171622 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.171289    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-54d4c4-dockercfg-t9ktx\""
Oct 30 19:56:55.175392 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.175342    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"]
Oct 30 19:56:55.203788 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.203751    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e3e551-15ee-4132-ad82-374dfc644e8d" path="/var/lib/kubelet/pods/d5e3e551-15ee-4132-ad82-374dfc644e8d/volumes"
Oct 30 19:56:55.204458 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.204436    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"]
Oct 30 19:56:55.237718 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.232874    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4swbm\" (UniqueName: \"kubernetes.io/projected/a414b711-860e-4c19-be75-4c1c1abc70b8-kube-api-access-4swbm\") pod \"payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q\" (UID: \"a414b711-860e-4c19-be75-4c1c1abc70b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"
Oct 30 19:56:55.237718 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.233474    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a414b711-860e-4c19-be75-4c1c1abc70b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q\" (UID: \"a414b711-860e-4c19-be75-4c1c1abc70b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"
Oct 30 19:56:55.237718 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.237512    6030 generic.go:356] "Generic (PLEG): container finished" podID="755da169-322e-451f-b4ac-90053aa2b956" containerID="5e8f0f8be5f0ef3befa5fd647e1cbb841fd1dbee7bad264ae65ebf6813325860" exitCode=0
Oct 30 19:56:55.237718 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.237631    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"
Oct 30 19:56:55.238133 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.238111    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"]
Oct 30 19:56:55.238543 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.238513    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"
Oct 30 19:56:55.239115 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.238870    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx" event={"ID":"755da169-322e-451f-b4ac-90053aa2b956","Type":"ContainerDied","Data":"5e8f0f8be5f0ef3befa5fd647e1cbb841fd1dbee7bad264ae65ebf6813325860"}
Oct 30 19:56:55.240835 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.240811    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx" event={"ID":"755da169-322e-451f-b4ac-90053aa2b956","Type":"ContainerDied","Data":"efa17639b0ccb55af3dcbae412fc6390dd5504886315551f4d96e82b8403dc21"}
Oct 30 19:56:55.241765 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.241143    6030 scope.go:122] "RemoveContainer" containerID="5e8f0f8be5f0ef3befa5fd647e1cbb841fd1dbee7bad264ae65ebf6813325860"
Oct 30 19:56:55.354120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.351340    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4swbm\" (UniqueName: \"kubernetes.io/projected/a414b711-860e-4c19-be75-4c1c1abc70b8-kube-api-access-4swbm\") pod \"payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q\" (UID: \"a414b711-860e-4c19-be75-4c1c1abc70b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"
Oct 30 19:56:55.354120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.351558    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a414b711-860e-4c19-be75-4c1c1abc70b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q\" (UID: \"a414b711-860e-4c19-be75-4c1c1abc70b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"
Oct 30 19:56:55.354120 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.352567    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a414b711-860e-4c19-be75-4c1c1abc70b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q\" (UID: \"a414b711-860e-4c19-be75-4c1c1abc70b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"
Oct 30 19:56:55.366323 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.366299    6030 scope.go:122] "RemoveContainer" containerID="5e8f0f8be5f0ef3befa5fd647e1cbb841fd1dbee7bad264ae65ebf6813325860"
Oct 30 19:56:55.366957 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:56:55.366933    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8f0f8be5f0ef3befa5fd647e1cbb841fd1dbee7bad264ae65ebf6813325860\": container with ID starting with 5e8f0f8be5f0ef3befa5fd647e1cbb841fd1dbee7bad264ae65ebf6813325860 not found: ID does not exist" containerID="5e8f0f8be5f0ef3befa5fd647e1cbb841fd1dbee7bad264ae65ebf6813325860"
Oct 30 19:56:55.367227 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.367158    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8f0f8be5f0ef3befa5fd647e1cbb841fd1dbee7bad264ae65ebf6813325860"} err="failed to get container status \"5e8f0f8be5f0ef3befa5fd647e1cbb841fd1dbee7bad264ae65ebf6813325860\": rpc error: code = NotFound desc = could not find container \"5e8f0f8be5f0ef3befa5fd647e1cbb841fd1dbee7bad264ae65ebf6813325860\": container with ID starting with 5e8f0f8be5f0ef3befa5fd647e1cbb841fd1dbee7bad264ae65ebf6813325860 not found: ID does not exist"
Oct 30 19:56:55.425173 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.420266    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4swbm\" (UniqueName: \"kubernetes.io/projected/a414b711-860e-4c19-be75-4c1c1abc70b8-kube-api-access-4swbm\") pod \"payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q\" (UID: \"a414b711-860e-4c19-be75-4c1c1abc70b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"
Oct 30 19:56:55.438515 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.438372    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"]
Oct 30 19:56:55.441514 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.441477    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-927bf8-9fb9d59bc-gkfmx"]
Oct 30 19:56:55.452544 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.452514    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68\" (UID: \"f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"
Oct 30 19:56:55.452701 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.452570    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4rwk\" (UniqueName: \"kubernetes.io/projected/f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4-kube-api-access-v4rwk\") pod \"payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68\" (UID: \"f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"
Oct 30 19:56:55.479908 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.479877    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"
Oct 30 19:56:55.555522 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.553758    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68\" (UID: \"f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"
Oct 30 19:56:55.555522 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.553893    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rwk\" (UniqueName: \"kubernetes.io/projected/f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4-kube-api-access-v4rwk\") pod \"payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68\" (UID: \"f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"
Oct 30 19:56:55.555522 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.555275    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68\" (UID: \"f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"
Oct 30 19:56:55.570447 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.570417    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4rwk\" (UniqueName: \"kubernetes.io/projected/f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4-kube-api-access-v4rwk\") pod \"payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68\" (UID: \"f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"
Oct 30 19:56:55.744989 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.744921    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"
Oct 30 19:56:55.773226 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:55.773201    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"]
Oct 30 19:56:55.777061 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:56:55.777006    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda414b711_860e_4c19_be75_4c1c1abc70b8.slice/crio-cc633dbcd6c35a93245ea561eee93d85e740bdeb5ab32e3c6a9443c005f295f3 WatchSource:0}: Error finding container cc633dbcd6c35a93245ea561eee93d85e740bdeb5ab32e3c6a9443c005f295f3: Status 404 returned error can't find the container with id cc633dbcd6c35a93245ea561eee93d85e740bdeb5ab32e3c6a9443c005f295f3
Oct 30 19:56:56.025067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.024969    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"]
Oct 30 19:56:56.029377 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.029353    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"
Oct 30 19:56:56.033703 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.033674    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"]
Oct 30 19:56:56.037727 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.037538    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"
Oct 30 19:56:56.038483 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.037870    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-779fa3-dockercfg-lxthv\""
Oct 30 19:56:56.038483 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.038442    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-779fa3\""
Oct 30 19:56:56.040648 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.040623    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"]
Oct 30 19:56:56.041408 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:56:56.041385    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35bc1c5_b72c_4e13_a28a_80b1f9e4bdf4.slice/crio-e93d9d60972c9a1aa66e96e1f8fd62db8eaabcbeb1f799d64aaac3b8d30803d1 WatchSource:0}: Error finding container e93d9d60972c9a1aa66e96e1f8fd62db8eaabcbeb1f799d64aaac3b8d30803d1: Status 404 returned error can't find the container with id e93d9d60972c9a1aa66e96e1f8fd62db8eaabcbeb1f799d64aaac3b8d30803d1
Oct 30 19:56:56.054070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.053975    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"]
Oct 30 19:56:56.059029 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.058995    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxptl\" (UniqueName: \"kubernetes.io/projected/40974280-ed25-41d5-8697-4c09d86aadb1-kube-api-access-sxptl\") pod \"payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8\" (UID: \"40974280-ed25-41d5-8697-4c09d86aadb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"
Oct 30 19:56:56.059211 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.059131    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpdrb\" (UniqueName: \"kubernetes.io/projected/1dfa9318-c8b7-4c06-91c5-f630b3dc7810-kube-api-access-tpdrb\") pod \"payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph\" (UID: \"1dfa9318-c8b7-4c06-91c5-f630b3dc7810\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"
Oct 30 19:56:56.059211 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.059200    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40974280-ed25-41d5-8697-4c09d86aadb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8\" (UID: \"40974280-ed25-41d5-8697-4c09d86aadb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"
Oct 30 19:56:56.059510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.059312    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfa9318-c8b7-4c06-91c5-f630b3dc7810-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph\" (UID: \"1dfa9318-c8b7-4c06-91c5-f630b3dc7810\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"
Oct 30 19:56:56.065290 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.065268    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"]
Oct 30 19:56:56.168846 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.163426    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpdrb\" (UniqueName: \"kubernetes.io/projected/1dfa9318-c8b7-4c06-91c5-f630b3dc7810-kube-api-access-tpdrb\") pod \"payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph\" (UID: \"1dfa9318-c8b7-4c06-91c5-f630b3dc7810\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"
Oct 30 19:56:56.212072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.208450    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40974280-ed25-41d5-8697-4c09d86aadb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8\" (UID: \"40974280-ed25-41d5-8697-4c09d86aadb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"
Oct 30 19:56:56.212072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.208611    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfa9318-c8b7-4c06-91c5-f630b3dc7810-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph\" (UID: \"1dfa9318-c8b7-4c06-91c5-f630b3dc7810\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"
Oct 30 19:56:56.212072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.208741    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxptl\" (UniqueName: \"kubernetes.io/projected/40974280-ed25-41d5-8697-4c09d86aadb1-kube-api-access-sxptl\") pod \"payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8\" (UID: \"40974280-ed25-41d5-8697-4c09d86aadb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"
Oct 30 19:56:56.212072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.202774    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpdrb\" (UniqueName: \"kubernetes.io/projected/1dfa9318-c8b7-4c06-91c5-f630b3dc7810-kube-api-access-tpdrb\") pod \"payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph\" (UID: \"1dfa9318-c8b7-4c06-91c5-f630b3dc7810\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"
Oct 30 19:56:56.212072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.209943    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40974280-ed25-41d5-8697-4c09d86aadb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8\" (UID: \"40974280-ed25-41d5-8697-4c09d86aadb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"
Oct 30 19:56:56.212072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.210608    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfa9318-c8b7-4c06-91c5-f630b3dc7810-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph\" (UID: \"1dfa9318-c8b7-4c06-91c5-f630b3dc7810\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"
Oct 30 19:56:56.314070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.313933    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68" event={"ID":"f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4","Type":"ContainerStarted","Data":"fdcb336a5775068b24f226049a2e95ba9c5995a06d9ba05fad312182afb11355"}
Oct 30 19:56:56.314564 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.314406    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68" event={"ID":"f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4","Type":"ContainerStarted","Data":"e93d9d60972c9a1aa66e96e1f8fd62db8eaabcbeb1f799d64aaac3b8d30803d1"}
Oct 30 19:56:56.315710 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.315651    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"
Oct 30 19:56:56.322720 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.321988    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q" event={"ID":"a414b711-860e-4c19-be75-4c1c1abc70b8","Type":"ContainerStarted","Data":"9603e6d1bac62e61b2e1b5ebccbc5f585a73d2f937c42cea9514c3d525b25397"}
Oct 30 19:56:56.322720 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.322060    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q" event={"ID":"a414b711-860e-4c19-be75-4c1c1abc70b8","Type":"ContainerStarted","Data":"cc633dbcd6c35a93245ea561eee93d85e740bdeb5ab32e3c6a9443c005f295f3"}
Oct 30 19:56:56.322720 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.322664    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"
Oct 30 19:56:56.325555 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.325533    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxptl\" (UniqueName: \"kubernetes.io/projected/40974280-ed25-41d5-8697-4c09d86aadb1-kube-api-access-sxptl\") pod \"payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8\" (UID: \"40974280-ed25-41d5-8697-4c09d86aadb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"
Oct 30 19:56:56.348573 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.348290    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"
Oct 30 19:56:56.367079 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.367013    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"
Oct 30 19:56:56.417453 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.417370    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68" podStartSLOduration=1.417348691 podStartE2EDuration="1.417348691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:56.366443548 +0000 UTC m=+1495.282050405" watchObservedRunningTime="2026-10-30 19:56:56.417348691 +0000 UTC m=+1495.332955550"
Oct 30 19:56:56.421178 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.421106    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q" podStartSLOduration=1.421093133 podStartE2EDuration="1.421093133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:56.4108158 +0000 UTC m=+1495.326422657" watchObservedRunningTime="2026-10-30 19:56:56.421093133 +0000 UTC m=+1495.336699988"
Oct 30 19:56:56.681302 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.681273    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"]
Oct 30 19:56:56.682017 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:56:56.681993    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40974280_ed25_41d5_8697_4c09d86aadb1.slice/crio-39b0a163815706f9e9bd467317ee4923f8d78e0087d63d8737979bdf1c470ad4 WatchSource:0}: Error finding container 39b0a163815706f9e9bd467317ee4923f8d78e0087d63d8737979bdf1c470ad4: Status 404 returned error can't find the container with id 39b0a163815706f9e9bd467317ee4923f8d78e0087d63d8737979bdf1c470ad4
Oct 30 19:56:56.702730 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:56:56.702702    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dfa9318_c8b7_4c06_91c5_f630b3dc7810.slice/crio-fe9d0702ea339e4020f00aeff2913e4cfb2b5d55e6d12a7ccd66f542b36cd598 WatchSource:0}: Error finding container fe9d0702ea339e4020f00aeff2913e4cfb2b5d55e6d12a7ccd66f542b36cd598: Status 404 returned error can't find the container with id fe9d0702ea339e4020f00aeff2913e4cfb2b5d55e6d12a7ccd66f542b36cd598
Oct 30 19:56:56.704919 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:56.704889    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"]
Oct 30 19:56:57.201490 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:57.201457    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="755da169-322e-451f-b4ac-90053aa2b956" path="/var/lib/kubelet/pods/755da169-322e-451f-b4ac-90053aa2b956/volumes"
Oct 30 19:56:57.329875 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:57.329839    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph" event={"ID":"1dfa9318-c8b7-4c06-91c5-f630b3dc7810","Type":"ContainerStarted","Data":"7b124f7a513a4e71d4037e30a62a911a8cb2695e51a7f57738d379933bc43d1a"}
Oct 30 19:56:57.330091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:57.329901    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"
Oct 30 19:56:57.330091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:57.329921    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph" event={"ID":"1dfa9318-c8b7-4c06-91c5-f630b3dc7810","Type":"ContainerStarted","Data":"fe9d0702ea339e4020f00aeff2913e4cfb2b5d55e6d12a7ccd66f542b36cd598"}
Oct 30 19:56:57.331492 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:57.331461    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8" event={"ID":"40974280-ed25-41d5-8697-4c09d86aadb1","Type":"ContainerStarted","Data":"b98c2d919b7f4e0e66f3469c1b01e04fab609ceb8c22748f58e764f6657082c6"}
Oct 30 19:56:57.331603 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:57.331519    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"
Oct 30 19:56:57.331603 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:57.331541    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8" event={"ID":"40974280-ed25-41d5-8697-4c09d86aadb1","Type":"ContainerStarted","Data":"39b0a163815706f9e9bd467317ee4923f8d78e0087d63d8737979bdf1c470ad4"}
Oct 30 19:56:57.360635 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:57.360576    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph" podStartSLOduration=1.3605586619999999 podStartE2EDuration="1.360558662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:57.357758417 +0000 UTC m=+1496.273365274" watchObservedRunningTime="2026-10-30 19:56:57.360558662 +0000 UTC m=+1496.276165518"
Oct 30 19:56:57.386887 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:57.386838    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8" podStartSLOduration=2.386823774 podStartE2EDuration="2.386823774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:57.384195772 +0000 UTC m=+1496.299802629" watchObservedRunningTime="2026-10-30 19:56:57.386823774 +0000 UTC m=+1496.302430633"
Oct 30 19:56:59.967867 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:59.967835    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"]
Oct 30 19:56:59.972193 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:59.972174    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 19:56:59.984517 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:59.984490    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"]
Oct 30 19:56:59.984779 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:56:59.984762    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-7eaeed-serving-cert\""
Oct 30 19:57:00.065830 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:00.065798    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7bc2a39-e277-402d-9d0f-c71910d82058-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg\" (UID: \"c7bc2a39-e277-402d-9d0f-c71910d82058\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 19:57:00.066001 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:00.065957    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7bc2a39-e277-402d-9d0f-c71910d82058-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg\" (UID: \"c7bc2a39-e277-402d-9d0f-c71910d82058\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 19:57:00.066159 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:00.066009    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w85n\" (UniqueName: \"kubernetes.io/projected/c7bc2a39-e277-402d-9d0f-c71910d82058-kube-api-access-8w85n\") pod \"maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg\" (UID: \"c7bc2a39-e277-402d-9d0f-c71910d82058\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 19:57:00.066159 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:00.066123    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7bc2a39-e277-402d-9d0f-c71910d82058-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg\" (UID: \"c7bc2a39-e277-402d-9d0f-c71910d82058\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 19:57:00.167340 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:00.167308    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7bc2a39-e277-402d-9d0f-c71910d82058-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg\" (UID: \"c7bc2a39-e277-402d-9d0f-c71910d82058\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 19:57:00.167515 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:00.167368    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w85n\" (UniqueName: \"kubernetes.io/projected/c7bc2a39-e277-402d-9d0f-c71910d82058-kube-api-access-8w85n\") pod \"maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg\" (UID: \"c7bc2a39-e277-402d-9d0f-c71910d82058\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 19:57:00.167515 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:00.167421    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7bc2a39-e277-402d-9d0f-c71910d82058-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg\" (UID: \"c7bc2a39-e277-402d-9d0f-c71910d82058\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 19:57:00.167515 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:00.167495    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7bc2a39-e277-402d-9d0f-c71910d82058-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg\" (UID: \"c7bc2a39-e277-402d-9d0f-c71910d82058\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 19:57:00.167644 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:57:00.167602    6030 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-serving-cert: secret "maas-api-e2e-shared-a-w1-7eaeed-serving-cert" not found
Oct 30 19:57:00.167682 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:57:00.167671    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7bc2a39-e277-402d-9d0f-c71910d82058-maas-api-tls podName:c7bc2a39-e277-402d-9d0f-c71910d82058 nodeName:}" failed. No retries permitted until 2026-10-30 19:57:00.667651653 +0000 UTC m=+1499.583258489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7bc2a39-e277-402d-9d0f-c71910d82058-maas-api-tls") pod "maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg" (UID: "c7bc2a39-e277-402d-9d0f-c71910d82058") : secret "maas-api-e2e-shared-a-w1-7eaeed-serving-cert" not found
Oct 30 19:57:00.168033 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:00.168015    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7bc2a39-e277-402d-9d0f-c71910d82058-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg\" (UID: \"c7bc2a39-e277-402d-9d0f-c71910d82058\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 19:57:00.170389 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:00.170362    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7bc2a39-e277-402d-9d0f-c71910d82058-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg\" (UID: \"c7bc2a39-e277-402d-9d0f-c71910d82058\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 19:57:00.182009 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:00.181981    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w85n\" (UniqueName: \"kubernetes.io/projected/c7bc2a39-e277-402d-9d0f-c71910d82058-kube-api-access-8w85n\") pod \"maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg\" (UID: \"c7bc2a39-e277-402d-9d0f-c71910d82058\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 19:57:00.673994 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:00.673948    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7bc2a39-e277-402d-9d0f-c71910d82058-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg\" (UID: \"c7bc2a39-e277-402d-9d0f-c71910d82058\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 19:57:00.689994 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:00.689962    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7bc2a39-e277-402d-9d0f-c71910d82058-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg\" (UID: \"c7bc2a39-e277-402d-9d0f-c71910d82058\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 19:57:00.892080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:00.892035    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 19:57:01.186478 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:01.186448    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"]
Oct 30 19:57:01.187451 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:57:01.187425    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7bc2a39_e277_402d_9d0f_c71910d82058.slice/crio-dd00af6d5f96afa7419a3f04280feedbf349932ec4cdd7a56dc1afbd240d3db7 WatchSource:0}: Error finding container dd00af6d5f96afa7419a3f04280feedbf349932ec4cdd7a56dc1afbd240d3db7: Status 404 returned error can't find the container with id dd00af6d5f96afa7419a3f04280feedbf349932ec4cdd7a56dc1afbd240d3db7
Oct 30 19:57:01.399216 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:01.396746    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg" event={"ID":"c7bc2a39-e277-402d-9d0f-c71910d82058","Type":"ContainerStarted","Data":"dd00af6d5f96afa7419a3f04280feedbf349932ec4cdd7a56dc1afbd240d3db7"}
Oct 30 19:57:02.403867 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:02.403830    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg" event={"ID":"c7bc2a39-e277-402d-9d0f-c71910d82058","Type":"ContainerStarted","Data":"97833b4a1d844b294a67ab8aec23f9a2fad45f9bacd18467112fe68a3ab17f3b"}
Oct 30 19:57:02.404333 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:02.403895    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 19:57:02.429885 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:02.429789    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg" podStartSLOduration=2.998447058 podStartE2EDuration="3.429774775s" totalImagesPullingTime="431.327717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:02.427580772 +0000 UTC m=+1501.343187629" watchObservedRunningTime="2026-10-30 19:57:02.429774775 +0000 UTC m=+1501.345381631"
Oct 30 19:57:03.167612 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:03.167568    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"]
Oct 30 19:57:03.167796 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:03.167775    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz" podUID="a29ff919-34f3-412f-adf8-d331698289e9" containerName="istio-proxy" containerID="cri-o://0494cffa8329d3f0cbf5d86a1393fad15a3105dcf35f943ef00aed5a79aa0a50" gracePeriod=30
Oct 30 19:57:03.205301 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:03.205264    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"
Oct 30 19:57:03.205470 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:03.205379    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"
Oct 30 19:57:04.293022 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:57:04.292985    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:57:05.208418 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:05.208380    6030 patch_prober.go:29] interesting pod/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:57:05.208601 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:05.208449    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz" podUID="a29ff919-34f3-412f-adf8-d331698289e9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:57:08.346573 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:08.346354    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"
Oct 30 19:57:08.346573 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:08.346474    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"
Oct 30 19:57:08.347873 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:08.347611    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"
Oct 30 19:57:08.348193 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:08.348157    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"
Oct 30 19:57:08.508680 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:08.502195    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 19:57:08.550106 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:08.550071    6030 generic.go:356] "Generic (PLEG): container finished" podID="a29ff919-34f3-412f-adf8-d331698289e9" containerID="0494cffa8329d3f0cbf5d86a1393fad15a3105dcf35f943ef00aed5a79aa0a50" exitCode=0
Oct 30 19:57:08.550285 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:08.550109    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz" event={"ID":"a29ff919-34f3-412f-adf8-d331698289e9","Type":"ContainerDied","Data":"0494cffa8329d3f0cbf5d86a1393fad15a3105dcf35f943ef00aed5a79aa0a50"}
Oct 30 19:57:08.550285 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:08.550167    6030 scope.go:122] "RemoveContainer" containerID="0c711057cba171197487995af0ab346ee6fbe181ca195025e5cece206601db4a"
Oct 30 19:57:08.856996 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:08.856913    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:57:09.034080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.027498    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-credential-socket\") pod \"a29ff919-34f3-412f-adf8-d331698289e9\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") "
Oct 30 19:57:09.034080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.027574    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-istio-data\") pod \"a29ff919-34f3-412f-adf8-d331698289e9\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") "
Oct 30 19:57:09.034080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.027612    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-workload-socket\") pod \"a29ff919-34f3-412f-adf8-d331698289e9\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") "
Oct 30 19:57:09.034080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.027663    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29ff919-34f3-412f-adf8-d331698289e9-istio-token\" (UniqueName: \"kubernetes.io/projected/a29ff919-34f3-412f-adf8-d331698289e9-istio-token\") pod \"a29ff919-34f3-412f-adf8-d331698289e9\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") "
Oct 30 19:57:09.034080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.027729    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-workload-certs\") pod \"a29ff919-34f3-412f-adf8-d331698289e9\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") "
Oct 30 19:57:09.034080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.027811    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a29ff919-34f3-412f-adf8-d331698289e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29ff919-34f3-412f-adf8-d331698289e9-istio-podinfo\") pod \"a29ff919-34f3-412f-adf8-d331698289e9\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") "
Oct 30 19:57:09.034080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.027907    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29ff919-34f3-412f-adf8-d331698289e9-kube-api-access-87xsd\" (UniqueName: \"kubernetes.io/projected/a29ff919-34f3-412f-adf8-d331698289e9-kube-api-access-87xsd\") pod \"a29ff919-34f3-412f-adf8-d331698289e9\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") "
Oct 30 19:57:09.034080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.027960    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29ff919-34f3-412f-adf8-d331698289e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29ff919-34f3-412f-adf8-d331698289e9-istiod-ca-cert\") pod \"a29ff919-34f3-412f-adf8-d331698289e9\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") "
Oct 30 19:57:09.034080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.027992    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-istio-envoy\") pod \"a29ff919-34f3-412f-adf8-d331698289e9\" (UID: \"a29ff919-34f3-412f-adf8-d331698289e9\") "
Oct 30 19:57:09.048766 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.048560    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-istio-data" pod "a29ff919-34f3-412f-adf8-d331698289e9" (UID: "a29ff919-34f3-412f-adf8-d331698289e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:09.050108 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.049003    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-credential-socket" pod "a29ff919-34f3-412f-adf8-d331698289e9" (UID: "a29ff919-34f3-412f-adf8-d331698289e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:09.050790 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.050678    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a29ff919-34f3-412f-adf8-d331698289e9-istio-podinfo" pod "a29ff919-34f3-412f-adf8-d331698289e9" (UID: "a29ff919-34f3-412f-adf8-d331698289e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:57:09.051183 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.051147    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-workload-socket" pod "a29ff919-34f3-412f-adf8-d331698289e9" (UID: "a29ff919-34f3-412f-adf8-d331698289e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:09.051332 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.051295    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29ff919-34f3-412f-adf8-d331698289e9-istiod-ca-cert" pod "a29ff919-34f3-412f-adf8-d331698289e9" (UID: "a29ff919-34f3-412f-adf8-d331698289e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:09.051688 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.051659    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-workload-certs" pod "a29ff919-34f3-412f-adf8-d331698289e9" (UID: "a29ff919-34f3-412f-adf8-d331698289e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:09.058078 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.056064    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29ff919-34f3-412f-adf8-d331698289e9-kube-api-access-87xsd" pod "a29ff919-34f3-412f-adf8-d331698289e9" (UID: "a29ff919-34f3-412f-adf8-d331698289e9"). InnerVolumeSpecName "kube-api-access-87xsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:09.058078 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.056427    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-istio-envoy" pod "a29ff919-34f3-412f-adf8-d331698289e9" (UID: "a29ff919-34f3-412f-adf8-d331698289e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:09.062864 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.062171    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29ff919-34f3-412f-adf8-d331698289e9-istio-token" pod "a29ff919-34f3-412f-adf8-d331698289e9" (UID: "a29ff919-34f3-412f-adf8-d331698289e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:09.130674 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.130639    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:09.130674 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.130677    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:09.130885 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.130693    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a29ff919-34f3-412f-adf8-d331698289e9-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:09.130885 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.130708    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:09.130885 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.130721    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29ff919-34f3-412f-adf8-d331698289e9-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:09.130885 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.130735    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87xsd\" (UniqueName: \"kubernetes.io/projected/a29ff919-34f3-412f-adf8-d331698289e9-kube-api-access-87xsd\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:09.130885 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.130749    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29ff919-34f3-412f-adf8-d331698289e9-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:09.130885 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.130762    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:09.130885 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.130784    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29ff919-34f3-412f-adf8-d331698289e9-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:09.558542 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.558466    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz" event={"ID":"a29ff919-34f3-412f-adf8-d331698289e9","Type":"ContainerDied","Data":"c55e66a939111b1e4b2f096eb22639a60bacfd97f84df1c97a8c8e3a3de28b3b"}
Oct 30 19:57:09.558542 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.558523    6030 scope.go:122] "RemoveContainer" containerID="0494cffa8329d3f0cbf5d86a1393fad15a3105dcf35f943ef00aed5a79aa0a50"
Oct 30 19:57:09.558542 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.558521    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"
Oct 30 19:57:09.601071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.599860    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"]
Oct 30 19:57:09.603785 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:09.603760    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-927bf8-openshift-default-6fd9c579c-8ljrz"]
Oct 30 19:57:11.206237 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:11.206200    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29ff919-34f3-412f-adf8-d331698289e9" path="/var/lib/kubelet/pods/a29ff919-34f3-412f-adf8-d331698289e9/volumes"
Oct 30 19:57:14.199075 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.192119    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"]
Oct 30 19:57:14.199075 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.193276    6030 cpu_manager.go:414] "containerMap: removing container" podUID="a29ff919-34f3-412f-adf8-d331698289e9" containerName="istio-proxy"
Oct 30 19:57:14.199075 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.193301    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29ff919-34f3-412f-adf8-d331698289e9" containerName="istio-proxy"
Oct 30 19:57:14.199075 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.193515    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29ff919-34f3-412f-adf8-d331698289e9" containerName="istio-proxy"
Oct 30 19:57:14.257146 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.245666    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"]
Oct 30 19:57:14.257146 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.247079    6030 cpu_manager.go:414] "containerMap: removing container" podUID="a29ff919-34f3-412f-adf8-d331698289e9" containerName="istio-proxy"
Oct 30 19:57:14.257146 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.247100    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29ff919-34f3-412f-adf8-d331698289e9" containerName="istio-proxy"
Oct 30 19:57:14.257146 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.247402    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29ff919-34f3-412f-adf8-d331698289e9" containerName="istio-proxy"
Oct 30 19:57:14.287125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.262161    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"
Oct 30 19:57:14.287125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.284863    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-7cbcbb\""
Oct 30 19:57:14.287125 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.285144    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-7cbcbb-dockercfg-6tjz8\""
Oct 30 19:57:14.338584 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.338554    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"]
Oct 30 19:57:14.338845 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.338828    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"]
Oct 30 19:57:14.338977 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.338840    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"
Oct 30 19:57:14.362958 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.362920    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e2a5df-6430-4eab-8959-ac0ca5a710ea-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4\" (UID: \"91e2a5df-6430-4eab-8959-ac0ca5a710ea\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"
Oct 30 19:57:14.363619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.363208    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhxsk\" (UniqueName: \"kubernetes.io/projected/91e2a5df-6430-4eab-8959-ac0ca5a710ea-kube-api-access-xhxsk\") pod \"payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4\" (UID: \"91e2a5df-6430-4eab-8959-ac0ca5a710ea\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"
Oct 30 19:57:14.464848 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.464813    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71903da-bab1-4a53-9576-a7357dfd22df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w\" (UID: \"b71903da-bab1-4a53-9576-a7357dfd22df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"
Oct 30 19:57:14.465057 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.465000    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e2a5df-6430-4eab-8959-ac0ca5a710ea-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4\" (UID: \"91e2a5df-6430-4eab-8959-ac0ca5a710ea\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"
Oct 30 19:57:14.465140 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.465107    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5zjh\" (UniqueName: \"kubernetes.io/projected/b71903da-bab1-4a53-9576-a7357dfd22df-kube-api-access-t5zjh\") pod \"payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w\" (UID: \"b71903da-bab1-4a53-9576-a7357dfd22df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"
Oct 30 19:57:14.465202 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.465148    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxsk\" (UniqueName: \"kubernetes.io/projected/91e2a5df-6430-4eab-8959-ac0ca5a710ea-kube-api-access-xhxsk\") pod \"payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4\" (UID: \"91e2a5df-6430-4eab-8959-ac0ca5a710ea\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"
Oct 30 19:57:14.465674 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.465656    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e2a5df-6430-4eab-8959-ac0ca5a710ea-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4\" (UID: \"91e2a5df-6430-4eab-8959-ac0ca5a710ea\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"
Oct 30 19:57:14.478996 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.478962    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhxsk\" (UniqueName: \"kubernetes.io/projected/91e2a5df-6430-4eab-8959-ac0ca5a710ea-kube-api-access-xhxsk\") pod \"payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4\" (UID: \"91e2a5df-6430-4eab-8959-ac0ca5a710ea\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"
Oct 30 19:57:14.566712 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.566560    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5zjh\" (UniqueName: \"kubernetes.io/projected/b71903da-bab1-4a53-9576-a7357dfd22df-kube-api-access-t5zjh\") pod \"payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w\" (UID: \"b71903da-bab1-4a53-9576-a7357dfd22df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"
Oct 30 19:57:14.566913 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.566742    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71903da-bab1-4a53-9576-a7357dfd22df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w\" (UID: \"b71903da-bab1-4a53-9576-a7357dfd22df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"
Oct 30 19:57:14.567417 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.567398    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71903da-bab1-4a53-9576-a7357dfd22df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w\" (UID: \"b71903da-bab1-4a53-9576-a7357dfd22df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"
Oct 30 19:57:14.588240 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.588214    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5zjh\" (UniqueName: \"kubernetes.io/projected/b71903da-bab1-4a53-9576-a7357dfd22df-kube-api-access-t5zjh\") pod \"payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w\" (UID: \"b71903da-bab1-4a53-9576-a7357dfd22df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"
Oct 30 19:57:14.645849 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.645813    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"
Oct 30 19:57:14.666648 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:14.666617    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"
Oct 30 19:57:15.071852 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:15.071745    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"]
Oct 30 19:57:15.083653 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:15.083274    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"]
Oct 30 19:57:15.628113 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:15.628071    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4" event={"ID":"91e2a5df-6430-4eab-8959-ac0ca5a710ea","Type":"ContainerStarted","Data":"fecd2c03b706292ec95cfe8731a520c8eb97cd2f2c63c3da55a23f8373dd5bff"}
Oct 30 19:57:15.628614 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:15.628140    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4" event={"ID":"91e2a5df-6430-4eab-8959-ac0ca5a710ea","Type":"ContainerStarted","Data":"3e6a93b3aaa311b276e6e171c738c4038c295e84557ce6aec5dfb75ea37e6131"}
Oct 30 19:57:15.628614 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:15.628272    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"
Oct 30 19:57:15.629927 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:15.629894    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w" event={"ID":"b71903da-bab1-4a53-9576-a7357dfd22df","Type":"ContainerStarted","Data":"803991ff564131928a97dce23bc13b42b6d31e37e4bd6d0ae95336f89dce134a"}
Oct 30 19:57:15.630095 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:15.629945    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w" event={"ID":"b71903da-bab1-4a53-9576-a7357dfd22df","Type":"ContainerStarted","Data":"4eee9765c69ad8aaa1e00e30efab81233f9b16105cc527ec44e4f55404d19cc6"}
Oct 30 19:57:15.630095 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:15.629973    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"
Oct 30 19:57:15.656616 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:15.656565    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4" podStartSLOduration=1.6565505900000002 podStartE2EDuration="1.65655059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:15.655088372 +0000 UTC m=+1514.570695232" watchObservedRunningTime="2026-10-30 19:57:15.65655059 +0000 UTC m=+1514.572157459"
Oct 30 19:57:15.679474 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:15.679423    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w" podStartSLOduration=1.6794055239999999 podStartE2EDuration="1.679405524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:15.675790012 +0000 UTC m=+1514.591396870" watchObservedRunningTime="2026-10-30 19:57:15.679405524 +0000 UTC m=+1514.595012381"
Oct 30 19:57:18.105363 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:18.104999    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 19:57:18.105363 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:18.105085    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm" podUID="c75bdfb0-89bd-4935-baae-55475fdb405a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 19:57:18.653731 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:18.653696    6030 generic.go:356] "Generic (PLEG): container finished" podID="c75bdfb0-89bd-4935-baae-55475fdb405a" containerID="80458fcd0987a05c5e049bbf172ce288143f933e723e9060d82b20e5789b24c2" exitCode=0
Oct 30 19:57:18.653911 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:18.653770    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm" event={"ID":"c75bdfb0-89bd-4935-baae-55475fdb405a","Type":"ContainerDied","Data":"80458fcd0987a05c5e049bbf172ce288143f933e723e9060d82b20e5789b24c2"}
Oct 30 19:57:18.654255 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:18.654237    6030 scope.go:122] "RemoveContainer" containerID="80458fcd0987a05c5e049bbf172ce288143f933e723e9060d82b20e5789b24c2"
Oct 30 19:57:18.655730 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:18.655696    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:57:18.655830 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:18.655785    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:57:18.655895 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:18.655835    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:57:19.661532 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:19.661503    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm" event={"ID":"c75bdfb0-89bd-4935-baae-55475fdb405a","Type":"ContainerStarted","Data":"be0e816709503de473df06f1d40d141a1ce92f3db035d05f848040644c047e18"}
Oct 30 19:57:20.206912 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:20.206875    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:57:20.207159 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:20.207120    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 19:57:20.207232 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:20.207187    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm" podUID="c75bdfb0-89bd-4935-baae-55475fdb405a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 19:57:21.173443 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:21.173411    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 19:57:21.174515 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:21.174482    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm" podUID="c75bdfb0-89bd-4935-baae-55475fdb405a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 19:57:22.176072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:22.175439    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 19:57:22.176072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:22.175515    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm" podUID="c75bdfb0-89bd-4935-baae-55475fdb405a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 19:57:23.176507 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:23.176470    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 19:57:23.177147 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:23.177113    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm" podUID="c75bdfb0-89bd-4935-baae-55475fdb405a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 19:57:24.190488 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:24.190452    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 19:57:24.191147 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:24.191114    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm" podUID="c75bdfb0-89bd-4935-baae-55475fdb405a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 19:57:25.178387 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:25.178349    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:57:25.178745 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:25.178724    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:57:25.179592 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:25.179567    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 19:57:26.501483 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.501438    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"]
Oct 30 19:57:26.506806 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.506780    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.517934 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.517903    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0eeba5\"/\"openshift-service-ca.crt\""
Oct 30 19:57:26.520180 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.520154    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0eeba5\"/\"kube-root-ca.crt\""
Oct 30 19:57:26.522621 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.522599    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"]
Oct 30 19:57:26.525580 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.525546    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0eeba5\"/\"auth-test-model-0eeba5-kserve-self-signed-certs\""
Oct 30 19:57:26.525711 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.525629    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0eeba5\"/\"default-dockercfg-822kd\""
Oct 30 19:57:26.554268 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.554226    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-home\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.554475 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.554355    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22dadc59-4ea6-4f6d-84e7-32735402c133-tls-certs\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.555225 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.555199    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-model-cache\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.555347 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.555272    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-dshm\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.555347 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.555312    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr9pn\" (UniqueName: \"kubernetes.io/projected/22dadc59-4ea6-4f6d-84e7-32735402c133-kube-api-access-qr9pn\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.555347 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.555341    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-tmp-dir\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.578248 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.578220    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"]
Oct 30 19:57:26.583401 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.583380    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"
Oct 30 19:57:26.604239 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.603550    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"]
Oct 30 19:57:26.617476 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.617450    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-981ee5\""
Oct 30 19:57:26.620623 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.620582    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-981ee5-dockercfg-sgjm6\""
Oct 30 19:57:26.643815 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.643788    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"
Oct 30 19:57:26.645833 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.645809    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"
Oct 30 19:57:26.656490 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.656461    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b8004a-deb0-418d-b6a9-bfd63122c373-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56\" (UID: \"f9b8004a-deb0-418d-b6a9-bfd63122c373\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"
Oct 30 19:57:26.656622 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.656538    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22dadc59-4ea6-4f6d-84e7-32735402c133-tls-certs\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.656738 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.656711    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-model-cache\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.656855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.656834    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-dshm\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.656922 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.656903    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr9pn\" (UniqueName: \"kubernetes.io/projected/22dadc59-4ea6-4f6d-84e7-32735402c133-kube-api-access-qr9pn\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.656981 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.656947    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwhtm\" (UniqueName: \"kubernetes.io/projected/f9b8004a-deb0-418d-b6a9-bfd63122c373-kube-api-access-fwhtm\") pod \"payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56\" (UID: \"f9b8004a-deb0-418d-b6a9-bfd63122c373\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"
Oct 30 19:57:26.657068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.656984    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-tmp-dir\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.657129 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.657090    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-home\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.657368 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.657344    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-tmp-dir\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.657456 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.657433    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-home\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.657536 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.657510    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-model-cache\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.659679 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.659652    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-dshm\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.659948 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.659927    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22dadc59-4ea6-4f6d-84e7-32735402c133-tls-certs\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.677744 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.677721    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr9pn\" (UniqueName: \"kubernetes.io/projected/22dadc59-4ea6-4f6d-84e7-32735402c133-kube-api-access-qr9pn\") pod \"auth-test-model-0eeba5-kserve-5f4d56485-r9p9w\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") " pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.758968 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.758370    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwhtm\" (UniqueName: \"kubernetes.io/projected/f9b8004a-deb0-418d-b6a9-bfd63122c373-kube-api-access-fwhtm\") pod \"payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56\" (UID: \"f9b8004a-deb0-418d-b6a9-bfd63122c373\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"
Oct 30 19:57:26.758968 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.758485    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b8004a-deb0-418d-b6a9-bfd63122c373-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56\" (UID: \"f9b8004a-deb0-418d-b6a9-bfd63122c373\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"
Oct 30 19:57:26.759490 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.759464    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b8004a-deb0-418d-b6a9-bfd63122c373-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56\" (UID: \"f9b8004a-deb0-418d-b6a9-bfd63122c373\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"
Oct 30 19:57:26.779603 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.779568    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwhtm\" (UniqueName: \"kubernetes.io/projected/f9b8004a-deb0-418d-b6a9-bfd63122c373-kube-api-access-fwhtm\") pod \"payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56\" (UID: \"f9b8004a-deb0-418d-b6a9-bfd63122c373\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"
Oct 30 19:57:26.838080 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.829335    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:26.933071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:26.920096    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"
Oct 30 19:57:27.385609 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:27.350384    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"]
Oct 30 19:57:27.385609 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:27.382061    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"]
Oct 30 19:57:27.425539 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:57:27.413552    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22dadc59_4ea6_4f6d_84e7_32735402c133.slice/crio-9113a841668556f7fa66d9f6a48680a17417104d847f23437e21810d1225152a WatchSource:0}: Error finding container 9113a841668556f7fa66d9f6a48680a17417104d847f23437e21810d1225152a: Status 404 returned error can't find the container with id 9113a841668556f7fa66d9f6a48680a17417104d847f23437e21810d1225152a
Oct 30 19:57:27.774388 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:27.774287    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w" event={"ID":"22dadc59-4ea6-4f6d-84e7-32735402c133","Type":"ContainerStarted","Data":"0e49588925c9fd60390197df12dc10f027d969ee8250efa7ad69bf950b85c0f4"}
Oct 30 19:57:27.774388 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:27.774340    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w" event={"ID":"22dadc59-4ea6-4f6d-84e7-32735402c133","Type":"ContainerStarted","Data":"9113a841668556f7fa66d9f6a48680a17417104d847f23437e21810d1225152a"}
Oct 30 19:57:27.776373 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:27.776334    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56" event={"ID":"f9b8004a-deb0-418d-b6a9-bfd63122c373","Type":"ContainerStarted","Data":"d21c8794f47892ab48997aad0894b74024b3fcd9f98b1f1a68309eea2f6400ba"}
Oct 30 19:57:27.776500 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:27.776381    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56" event={"ID":"f9b8004a-deb0-418d-b6a9-bfd63122c373","Type":"ContainerStarted","Data":"7d6106d270ab25258bb586143ba6be5b8c1b3f1e47e983849ebe6d6f0a626246"}
Oct 30 19:57:27.776500 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:27.776412    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"
Oct 30 19:57:27.832406 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:27.832258    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:27.875091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:27.875015    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56" podStartSLOduration=1.87499836 podStartE2EDuration="1.87499836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:27.873087358 +0000 UTC m=+1526.788694213" watchObservedRunningTime="2026-10-30 19:57:27.87499836 +0000 UTC m=+1526.790605210"
Oct 30 19:57:27.876022 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:27.875988    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w" podStartSLOduration=1.875977544 podStartE2EDuration="1.875977544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:27.826339544 +0000 UTC m=+1526.741946400" watchObservedRunningTime="2026-10-30 19:57:27.875977544 +0000 UTC m=+1526.791584400"
Oct 30 19:57:29.846233 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:29.846199    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"]
Oct 30 19:57:29.850488 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:29.850460    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:29.869417 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:29.869384    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-b24a1a\"/\"openshift-service-ca.crt\""
Oct 30 19:57:29.871216 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:29.871188    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-b24a1a\"/\"kube-root-ca.crt\""
Oct 30 19:57:29.871216 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:29.871188    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-b24a1a\"/\"auth-test-model-b24a1a-kserve-self-signed-certs\""
Oct 30 19:57:29.871216 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:29.871191    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-b24a1a\"/\"default-dockercfg-f9g47\""
Oct 30 19:57:29.872662 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:29.872635    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"]
Oct 30 19:57:29.932962 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:29.932924    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6bkw\" (UniqueName: \"kubernetes.io/projected/864c1436-736d-4921-b137-ed7d41914a74-kube-api-access-h6bkw\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:29.933173 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:29.933030    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-tmp-dir\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:29.933275 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:29.933169    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-dshm\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:29.933275 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:29.933203    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-model-cache\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:29.933391 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:29.933344    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864c1436-736d-4921-b137-ed7d41914a74-tls-certs\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:29.933541 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:29.933521    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-home\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:30.035280 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:30.035245    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6bkw\" (UniqueName: \"kubernetes.io/projected/864c1436-736d-4921-b137-ed7d41914a74-kube-api-access-h6bkw\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:30.035500 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:30.035287    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-tmp-dir\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:30.035500 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:30.035322    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-dshm\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:30.035500 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:30.035455    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-model-cache\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:30.035677 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:30.035552    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864c1436-736d-4921-b137-ed7d41914a74-tls-certs\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:30.035737 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:30.035709    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-home\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:30.035829 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:30.035712    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-tmp-dir\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:30.035890 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:30.035842    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-model-cache\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:30.036105 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:30.036083    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-home\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:30.037837 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:30.037806    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-dshm\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:30.038314 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:30.038297    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864c1436-736d-4921-b137-ed7d41914a74-tls-certs\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:30.048566 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:30.048541    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6bkw\" (UniqueName: \"kubernetes.io/projected/864c1436-736d-4921-b137-ed7d41914a74-kube-api-access-h6bkw\") pod \"auth-test-model-b24a1a-kserve-5f8b7fffb5-52725\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") " pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:30.179073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:30.165732    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:30.525353 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:30.525325    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"]
Oct 30 19:57:30.527162 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:57:30.527137    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod864c1436_736d_4921_b137_ed7d41914a74.slice/crio-66d97f0c905397619edddc295ec80011ae2cb1cc7cbea01638720281c38b1652 WatchSource:0}: Error finding container 66d97f0c905397619edddc295ec80011ae2cb1cc7cbea01638720281c38b1652: Status 404 returned error can't find the container with id 66d97f0c905397619edddc295ec80011ae2cb1cc7cbea01638720281c38b1652
Oct 30 19:57:30.797486 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:30.797340    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725" event={"ID":"864c1436-736d-4921-b137-ed7d41914a74","Type":"ContainerStarted","Data":"37777689b708c751660c389a9fc6aa23d69de38ccf066cd09d969fecfeb697e2"}
Oct 30 19:57:30.797486 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:30.797391    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725" event={"ID":"864c1436-736d-4921-b137-ed7d41914a74","Type":"ContainerStarted","Data":"66d97f0c905397619edddc295ec80011ae2cb1cc7cbea01638720281c38b1652"}
Oct 30 19:57:30.822354 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:30.822308    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725" podStartSLOduration=1.822294124 podStartE2EDuration="1.822294124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:30.821856491 +0000 UTC m=+1529.737463370" watchObservedRunningTime="2026-10-30 19:57:30.822294124 +0000 UTC m=+1529.737900980"
Oct 30 19:57:31.167165 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:31.167124    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:34.290311 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:57:34.290269    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:57:36.830619 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:36.830577    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:36.844002 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:36.843974    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:36.873235 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:36.873209    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 19:57:38.785369 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:38.785340    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"
Oct 30 19:57:40.166740 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:40.166705    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:40.180426 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:40.180399    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:57:40.917986 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:57:40.917901    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:58:02.201360 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.201334    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"]
Oct 30 19:58:02.208671 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.208650    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.219917 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.219885    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-b7431c\"/\"auth-test-model-b7431c-kserve-self-signed-certs\""
Oct 30 19:58:02.220083 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.220000    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-b7431c\"/\"default-dockercfg-ln5jq\""
Oct 30 19:58:02.222399 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.222373    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"]
Oct 30 19:58:02.228787 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.228766    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-b7431c\"/\"kube-root-ca.crt\""
Oct 30 19:58:02.229119 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.229030    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-b7431c\"/\"openshift-service-ca.crt\""
Oct 30 19:58:02.320478 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.320439    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-home\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.322134 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.322102    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc79q\" (UniqueName: \"kubernetes.io/projected/0d51a759-22ec-4891-b58e-c8b72e85fbcc-kube-api-access-lc79q\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.322355 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.322337    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-dshm\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.322510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.322494    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-tmp-dir\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.322840 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.322808    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-model-cache\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.322932 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.322902    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d51a759-22ec-4891-b58e-c8b72e85fbcc-tls-certs\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.424197 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.424155    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-tmp-dir\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.424391 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.424265    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-model-cache\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.424391 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.424310    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d51a759-22ec-4891-b58e-c8b72e85fbcc-tls-certs\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.424495 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.424469    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-home\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.424552 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.424537    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc79q\" (UniqueName: \"kubernetes.io/projected/0d51a759-22ec-4891-b58e-c8b72e85fbcc-kube-api-access-lc79q\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.424612 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.424581    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-dshm\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.424698 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.424671    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-tmp-dir\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.424801 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.424766    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-model-cache\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.425017 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.424994    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-home\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.427471 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.427449    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-dshm\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.427678 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.427657    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d51a759-22ec-4891-b58e-c8b72e85fbcc-tls-certs\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.442360 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.442335    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc79q\" (UniqueName: \"kubernetes.io/projected/0d51a759-22ec-4891-b58e-c8b72e85fbcc-kube-api-access-lc79q\") pod \"auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") " pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:02.524815 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:02.524324    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:03.003855 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:03.003826    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"]
Oct 30 19:58:03.005695 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:58:03.005631    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d51a759_22ec_4891_b58e_c8b72e85fbcc.slice/crio-61a60dfd5db25e17fbbd7615ed962a65b3b0ee335af8a0f95031980a310544d0 WatchSource:0}: Error finding container 61a60dfd5db25e17fbbd7615ed962a65b3b0ee335af8a0f95031980a310544d0: Status 404 returned error can't find the container with id 61a60dfd5db25e17fbbd7615ed962a65b3b0ee335af8a0f95031980a310544d0
Oct 30 19:58:03.172804 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:03.172769    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6" event={"ID":"0d51a759-22ec-4891-b58e-c8b72e85fbcc","Type":"ContainerStarted","Data":"1e113ebf471f91d0a86b4bc92fa343205599070ece9cb2b6cf39f17a456c14ea"}
Oct 30 19:58:03.173002 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:03.172812    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6" event={"ID":"0d51a759-22ec-4891-b58e-c8b72e85fbcc","Type":"ContainerStarted","Data":"61a60dfd5db25e17fbbd7615ed962a65b3b0ee335af8a0f95031980a310544d0"}
Oct 30 19:58:03.209963 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:03.209909    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6" podStartSLOduration=1.209895366 podStartE2EDuration="1.209895366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:03.206005196 +0000 UTC m=+1562.121612052" watchObservedRunningTime="2026-10-30 19:58:03.209895366 +0000 UTC m=+1562.125502226"
Oct 30 19:58:03.533114 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:03.526655    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:04.292825 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:58:04.292783    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:58:05.775959 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.775921    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"]
Oct 30 19:58:05.780715 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.780689    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:05.787970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.787944    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-7eaeed\"/\"openshift-service-ca.crt\""
Oct 30 19:58:05.789136 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.789112    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"]
Oct 30 19:58:05.792265 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.791674    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-7eaeed\"/\"kube-root-ca.crt\""
Oct 30 19:58:05.793462 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.793418    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-7eaeed\"/\"default-dockercfg-cx4bn\""
Oct 30 19:58:05.793596 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.793462    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-7eaeed\"/\"auth-test-model-7eaeed-kserve-self-signed-certs\""
Oct 30 19:58:05.868267 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.868228    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-tmp-dir\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:05.868491 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.868356    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h7fs\" (UniqueName: \"kubernetes.io/projected/5983703e-9048-445d-b9db-82e8bcff02a8-kube-api-access-6h7fs\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:05.868491 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.868395    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5983703e-9048-445d-b9db-82e8bcff02a8-tls-certs\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:05.868491 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.868444    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-dshm\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:05.868491 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.868485    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-home\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:05.868729 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.868540    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-model-cache\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:05.982067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.969895    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-model-cache\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:05.982067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.970942    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-model-cache\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:05.982067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.971020    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-tmp-dir\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:05.982067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.971161    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h7fs\" (UniqueName: \"kubernetes.io/projected/5983703e-9048-445d-b9db-82e8bcff02a8-kube-api-access-6h7fs\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:05.982067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.971200    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5983703e-9048-445d-b9db-82e8bcff02a8-tls-certs\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:05.982067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.971245    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-dshm\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:05.982067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.971285    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-home\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:05.982067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.971618    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-home\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:05.982067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.971831    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-tmp-dir\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:05.990562 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.990032    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-dshm\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:06.004805 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:05.998217    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h7fs\" (UniqueName: \"kubernetes.io/projected/5983703e-9048-445d-b9db-82e8bcff02a8-kube-api-access-6h7fs\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:06.018072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:06.014910    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5983703e-9048-445d-b9db-82e8bcff02a8-tls-certs\") pod \"auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") " pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:06.097837 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:06.097468    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:06.497293 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:06.497258    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"]
Oct 30 19:58:06.498373 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:58:06.498338    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5983703e_9048_445d_b9db_82e8bcff02a8.slice/crio-2c5347ade8098ff45136d3b0e2de09156c71914b195cad75ccf2b84094e5ef31 WatchSource:0}: Error finding container 2c5347ade8098ff45136d3b0e2de09156c71914b195cad75ccf2b84094e5ef31: Status 404 returned error can't find the container with id 2c5347ade8098ff45136d3b0e2de09156c71914b195cad75ccf2b84094e5ef31
Oct 30 19:58:07.258095 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:07.255495    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr" event={"ID":"5983703e-9048-445d-b9db-82e8bcff02a8","Type":"ContainerStarted","Data":"535f188f0a815d753aa91937b97e45fa6149bd7d036328f696433741b576ae87"}
Oct 30 19:58:07.258095 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:07.255550    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr" event={"ID":"5983703e-9048-445d-b9db-82e8bcff02a8","Type":"ContainerStarted","Data":"2c5347ade8098ff45136d3b0e2de09156c71914b195cad75ccf2b84094e5ef31"}
Oct 30 19:58:07.282813 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:07.282734    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr" podStartSLOduration=2.282718482 podStartE2EDuration="2.282718482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:07.282162778 +0000 UTC m=+1566.197769635" watchObservedRunningTime="2026-10-30 19:58:07.282718482 +0000 UTC m=+1566.198325378"
Oct 30 19:58:08.102320 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:08.098744    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:12.525107 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:12.525067    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:12.538300 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:12.538275    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:13.317019 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:13.316983    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 19:58:16.098509 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:16.098473    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:16.113500 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:16.113476    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:16.340165 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:16.340128    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:16.804674 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:16.804643    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"]
Oct 30 19:58:16.804910 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:16.804880    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm" podUID="8d665c73-057a-47b6-9b8c-76768771182f" containerName="istio-proxy" containerID="cri-o://1f7a34ebe781cd6099e72fd2f6bbca97eccab79d3d0994cb097f78059be91132" gracePeriod=30
Oct 30 19:58:16.912510 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:16.912476    6030 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:58:16.912703 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:16.912543    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm" podUID="8d665c73-057a-47b6-9b8c-76768771182f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:58:22.092628 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.092601    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:58:22.155335 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.155305    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-istio-envoy\") pod \"8d665c73-057a-47b6-9b8c-76768771182f\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") "
Oct 30 19:58:22.155648 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.155621    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d665c73-057a-47b6-9b8c-76768771182f-kube-api-access-wd2wk\" (UniqueName: \"kubernetes.io/projected/8d665c73-057a-47b6-9b8c-76768771182f-kube-api-access-wd2wk\") pod \"8d665c73-057a-47b6-9b8c-76768771182f\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") "
Oct 30 19:58:22.155832 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.155689    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-istio-data\") pod \"8d665c73-057a-47b6-9b8c-76768771182f\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") "
Oct 30 19:58:22.155832 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.155737    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d665c73-057a-47b6-9b8c-76768771182f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d665c73-057a-47b6-9b8c-76768771182f-istio-podinfo\") pod \"8d665c73-057a-47b6-9b8c-76768771182f\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") "
Oct 30 19:58:22.155832 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.155782    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d665c73-057a-47b6-9b8c-76768771182f-istio-token\" (UniqueName: \"kubernetes.io/projected/8d665c73-057a-47b6-9b8c-76768771182f-istio-token\") pod \"8d665c73-057a-47b6-9b8c-76768771182f\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") "
Oct 30 19:58:22.155832 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.155808    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d665c73-057a-47b6-9b8c-76768771182f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d665c73-057a-47b6-9b8c-76768771182f-istiod-ca-cert\") pod \"8d665c73-057a-47b6-9b8c-76768771182f\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") "
Oct 30 19:58:22.156117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.155871    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-workload-socket\") pod \"8d665c73-057a-47b6-9b8c-76768771182f\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") "
Oct 30 19:58:22.156117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.155956    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-credential-socket\") pod \"8d665c73-057a-47b6-9b8c-76768771182f\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") "
Oct 30 19:58:22.156117 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.155999    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-workload-certs\") pod \"8d665c73-057a-47b6-9b8c-76768771182f\" (UID: \"8d665c73-057a-47b6-9b8c-76768771182f\") "
Oct 30 19:58:22.156797 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.156770    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-workload-certs" pod "8d665c73-057a-47b6-9b8c-76768771182f" (UID: "8d665c73-057a-47b6-9b8c-76768771182f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:22.160100 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.158493    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d665c73-057a-47b6-9b8c-76768771182f-istiod-ca-cert" pod "8d665c73-057a-47b6-9b8c-76768771182f" (UID: "8d665c73-057a-47b6-9b8c-76768771182f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:22.160100 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.158675    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-workload-socket" pod "8d665c73-057a-47b6-9b8c-76768771182f" (UID: "8d665c73-057a-47b6-9b8c-76768771182f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:22.160100 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.158984    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-credential-socket" pod "8d665c73-057a-47b6-9b8c-76768771182f" (UID: "8d665c73-057a-47b6-9b8c-76768771182f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:22.160100 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.159399    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-istio-data" pod "8d665c73-057a-47b6-9b8c-76768771182f" (UID: "8d665c73-057a-47b6-9b8c-76768771182f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:22.160100 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.159713    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-istio-envoy" pod "8d665c73-057a-47b6-9b8c-76768771182f" (UID: "8d665c73-057a-47b6-9b8c-76768771182f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:22.160717 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.160691    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d665c73-057a-47b6-9b8c-76768771182f-istio-token" pod "8d665c73-057a-47b6-9b8c-76768771182f" (UID: "8d665c73-057a-47b6-9b8c-76768771182f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:22.161565 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.161539    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d665c73-057a-47b6-9b8c-76768771182f-kube-api-access-wd2wk" pod "8d665c73-057a-47b6-9b8c-76768771182f" (UID: "8d665c73-057a-47b6-9b8c-76768771182f"). InnerVolumeSpecName "kube-api-access-wd2wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:22.163517 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.163472    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d665c73-057a-47b6-9b8c-76768771182f-istio-podinfo" pod "8d665c73-057a-47b6-9b8c-76768771182f" (UID: "8d665c73-057a-47b6-9b8c-76768771182f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:58:22.258305 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.258264    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:22.258305 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.258300    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:22.258305 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.258315    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:22.258305 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.258329    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd2wk\" (UniqueName: \"kubernetes.io/projected/8d665c73-057a-47b6-9b8c-76768771182f-kube-api-access-wd2wk\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:22.258675 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.258342    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:22.258675 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.258355    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d665c73-057a-47b6-9b8c-76768771182f-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:22.258675 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.258367    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d665c73-057a-47b6-9b8c-76768771182f-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:22.258675 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.258379    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d665c73-057a-47b6-9b8c-76768771182f-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:22.258675 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.258423    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d665c73-057a-47b6-9b8c-76768771182f-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:22.369615 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.369343    6030 generic.go:356] "Generic (PLEG): container finished" podID="8d665c73-057a-47b6-9b8c-76768771182f" containerID="1f7a34ebe781cd6099e72fd2f6bbca97eccab79d3d0994cb097f78059be91132" exitCode=0
Oct 30 19:58:22.369615 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.369410    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm" event={"ID":"8d665c73-057a-47b6-9b8c-76768771182f","Type":"ContainerDied","Data":"1f7a34ebe781cd6099e72fd2f6bbca97eccab79d3d0994cb097f78059be91132"}
Oct 30 19:58:22.369615 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.369458    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm" event={"ID":"8d665c73-057a-47b6-9b8c-76768771182f","Type":"ContainerDied","Data":"05137ba69bd860177a1e8cbbfc07e376961b09e98c67c5ce0db1916adff7aa1f"}
Oct 30 19:58:22.369615 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.369485    6030 scope.go:122] "RemoveContainer" containerID="1f7a34ebe781cd6099e72fd2f6bbca97eccab79d3d0994cb097f78059be91132"
Oct 30 19:58:22.387260 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.369656    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"
Oct 30 19:58:22.500413 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.498229    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"]
Oct 30 19:58:22.500413 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.498320    6030 scope.go:122] "RemoveContainer" containerID="e0f9e8d327d5ec71302370a739a9c82e1b1ebd4eb442eea6dba5a3ada5bef393"
Oct 30 19:58:22.500413 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.499798    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-6l2jm"]
Oct 30 19:58:22.516550 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.516073    6030 scope.go:122] "RemoveContainer" containerID="1f7a34ebe781cd6099e72fd2f6bbca97eccab79d3d0994cb097f78059be91132"
Oct 30 19:58:22.516988 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:58:22.516753    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7a34ebe781cd6099e72fd2f6bbca97eccab79d3d0994cb097f78059be91132\": container with ID starting with 1f7a34ebe781cd6099e72fd2f6bbca97eccab79d3d0994cb097f78059be91132 not found: ID does not exist" containerID="1f7a34ebe781cd6099e72fd2f6bbca97eccab79d3d0994cb097f78059be91132"
Oct 30 19:58:22.516988 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.516782    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7a34ebe781cd6099e72fd2f6bbca97eccab79d3d0994cb097f78059be91132"} err="failed to get container status \"1f7a34ebe781cd6099e72fd2f6bbca97eccab79d3d0994cb097f78059be91132\": rpc error: code = NotFound desc = could not find container \"1f7a34ebe781cd6099e72fd2f6bbca97eccab79d3d0994cb097f78059be91132\": container with ID starting with 1f7a34ebe781cd6099e72fd2f6bbca97eccab79d3d0994cb097f78059be91132 not found: ID does not exist"
Oct 30 19:58:22.516988 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.516802    6030 scope.go:122] "RemoveContainer" containerID="e0f9e8d327d5ec71302370a739a9c82e1b1ebd4eb442eea6dba5a3ada5bef393"
Oct 30 19:58:22.517360 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:58:22.517271    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f9e8d327d5ec71302370a739a9c82e1b1ebd4eb442eea6dba5a3ada5bef393\": container with ID starting with e0f9e8d327d5ec71302370a739a9c82e1b1ebd4eb442eea6dba5a3ada5bef393 not found: ID does not exist" containerID="e0f9e8d327d5ec71302370a739a9c82e1b1ebd4eb442eea6dba5a3ada5bef393"
Oct 30 19:58:22.517360 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:22.517300    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f9e8d327d5ec71302370a739a9c82e1b1ebd4eb442eea6dba5a3ada5bef393"} err="failed to get container status \"e0f9e8d327d5ec71302370a739a9c82e1b1ebd4eb442eea6dba5a3ada5bef393\": rpc error: code = NotFound desc = could not find container \"e0f9e8d327d5ec71302370a739a9c82e1b1ebd4eb442eea6dba5a3ada5bef393\": container with ID starting with e0f9e8d327d5ec71302370a739a9c82e1b1ebd4eb442eea6dba5a3ada5bef393 not found: ID does not exist"
Oct 30 19:58:23.202553 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:23.202521    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d665c73-057a-47b6-9b8c-76768771182f" path="/var/lib/kubelet/pods/8d665c73-057a-47b6-9b8c-76768771182f/volumes"
Oct 30 19:58:28.539820 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.539782    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"]
Oct 30 19:58:28.540709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.540688    6030 cpu_manager.go:414] "containerMap: removing container" podUID="8d665c73-057a-47b6-9b8c-76768771182f" containerName="istio-proxy"
Oct 30 19:58:28.540755 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.540712    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d665c73-057a-47b6-9b8c-76768771182f" containerName="istio-proxy"
Oct 30 19:58:28.540926 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.540915    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d665c73-057a-47b6-9b8c-76768771182f" containerName="istio-proxy"
Oct 30 19:58:28.546266 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.546245    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.559097 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.559069    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"]
Oct 30 19:58:28.567451 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.567421    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"]
Oct 30 19:58:28.567997 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.567980    6030 cpu_manager.go:414] "containerMap: removing container" podUID="8d665c73-057a-47b6-9b8c-76768771182f" containerName="istio-proxy"
Oct 30 19:58:28.567997 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.567995    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d665c73-057a-47b6-9b8c-76768771182f" containerName="istio-proxy"
Oct 30 19:58:28.568231 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.568131    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d665c73-057a-47b6-9b8c-76768771182f" containerName="istio-proxy"
Oct 30 19:58:28.579414 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.579369    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.618067 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.608606    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"]
Oct 30 19:58:28.684697 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.684658    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505b8b25-660e-432f-87b0-f359078e9f2a-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.684697 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.684702    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.684914 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.684732    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmwmz\" (UniqueName: \"kubernetes.io/projected/505b8b25-660e-432f-87b0-f359078e9f2a-kube-api-access-cmwmz\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.684914 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.684843    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.684914 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.684886    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.684914 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.684907    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.685148 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.685022    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.685148 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.685091    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e688d9-46c5-47a1-8150-de9e8e024c49-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.685148 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.685124    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.685248 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.685222    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvz29\" (UniqueName: \"kubernetes.io/projected/57e688d9-46c5-47a1-8150-de9e8e024c49-kube-api-access-cvz29\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.685302 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.685286    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.685349 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.685316    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.685424 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.685408    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505b8b25-660e-432f-87b0-f359078e9f2a-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.685526 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.685513    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.685564 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.685540    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.685599 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.685583    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505b8b25-660e-432f-87b0-f359078e9f2a-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.685645 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.685633    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.685685 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.685660    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.786434 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.786400    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.786434 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.786438    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.786647 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.786485    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505b8b25-660e-432f-87b0-f359078e9f2a-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.786647 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.786508    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.786647 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.786531    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmwmz\" (UniqueName: \"kubernetes.io/projected/505b8b25-660e-432f-87b0-f359078e9f2a-kube-api-access-cmwmz\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.786843 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.786667    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.786843 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.786730    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.786843 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.786763    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.786843 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.786830    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.787091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.786880    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.787091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.786891    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.787091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.786940    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e688d9-46c5-47a1-8150-de9e8e024c49-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.787091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.786901    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.787091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.786973    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.787091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.787039    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvz29\" (UniqueName: \"kubernetes.io/projected/57e688d9-46c5-47a1-8150-de9e8e024c49-kube-api-access-cvz29\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.787091 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.787083    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.787709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.787143    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.787709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.787155    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.787709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.787174    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.787709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.787210    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505b8b25-660e-432f-87b0-f359078e9f2a-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.787709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.787247    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.787709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.787299    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.787709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.787313    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.787709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.787353    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.787709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.787410    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505b8b25-660e-432f-87b0-f359078e9f2a-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.787709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.787566    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.788287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.787718    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e688d9-46c5-47a1-8150-de9e8e024c49-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.788287 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.788039    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505b8b25-660e-432f-87b0-f359078e9f2a-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.789479 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.789453    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.789605 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.789530    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.790440 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.790391    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505b8b25-660e-432f-87b0-f359078e9f2a-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.790440 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.790391    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.802835 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.802749    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505b8b25-660e-432f-87b0-f359078e9f2a-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.803024 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.803003    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmwmz\" (UniqueName: \"kubernetes.io/projected/505b8b25-660e-432f-87b0-f359078e9f2a-kube-api-access-cmwmz\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.808526 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.808502    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.814474 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.814450    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvz29\" (UniqueName: \"kubernetes.io/projected/57e688d9-46c5-47a1-8150-de9e8e024c49-kube-api-access-cvz29\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.862519 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.862483    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:28.909999 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.909970    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:28.949363 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.947346    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"]
Oct 30 19:58:28.977143 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:28.975287    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.023466 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.022723    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b24a1a-openshift-default-dockercfg-h5n8n\""
Oct 30 19:58:29.030315 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.028934    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"]
Oct 30 19:58:29.032534 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.031572    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-credential-socket\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.032534 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.031606    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-workload-certs\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.032534 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.031741    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-workload-socket\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.133419 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.133383    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-envoy\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.133608 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.133449    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvpvs\" (UniqueName: \"kubernetes.io/projected/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-kube-api-access-qvpvs\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.133608 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.133545    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-data\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.133608 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.133591    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-workload-socket\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.133741 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.133639    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-token\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.133936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.133916    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-workload-socket\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.134483 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.134459    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istiod-ca-cert\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.134539 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.134518    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-podinfo\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.134598 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.134578    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-credential-socket\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.134642 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.134612    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-workload-certs\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.134944 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.134922    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-credential-socket\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.134982 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.134960    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-workload-certs\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.177682 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.177644    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"]
Oct 30 19:58:29.177791 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:58:29.177724    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505b8b25_660e_432f_87b0_f359078e9f2a.slice/crio-c8120bd38425fdeea9f04953e1e7d726e17a77b0882ea4c8becd993d2c4fdb22 WatchSource:0}: Error finding container c8120bd38425fdeea9f04953e1e7d726e17a77b0882ea4c8becd993d2c4fdb22: Status 404 returned error can't find the container with id c8120bd38425fdeea9f04953e1e7d726e17a77b0882ea4c8becd993d2c4fdb22
Oct 30 19:58:29.180492 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.180456    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:58:29.180607 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.180531    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:58:29.180607 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.180574    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:58:29.235522 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.235491    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istiod-ca-cert\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.235677 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.235557    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-podinfo\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.235749 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.235732    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-envoy\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.235818 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.235774    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvpvs\" (UniqueName: \"kubernetes.io/projected/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-kube-api-access-qvpvs\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.235818 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.235808    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-data\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.235930 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.235875    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-token\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.236175 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.236148    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-data\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.236293 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.236188    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istiod-ca-cert\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.238344 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.238317    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-envoy\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.238576 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.238558    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-podinfo\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.248970 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.248947    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvpvs\" (UniqueName: \"kubernetes.io/projected/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-kube-api-access-qvpvs\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.249164 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.249146    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-token\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.320020 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.319983    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"]
Oct 30 19:58:29.324927 ip-10-0-142-167 kubenswrapper[6030]: W1030 19:58:29.324897    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e688d9_46c5_47a1_8150_de9e8e024c49.slice/crio-212d43f70138f796c317c87ee263c13b9b672d9b91bc48cea7bb0fb1a8febf35 WatchSource:0}: Error finding container 212d43f70138f796c317c87ee263c13b9b672d9b91bc48cea7bb0fb1a8febf35: Status 404 returned error can't find the container with id 212d43f70138f796c317c87ee263c13b9b672d9b91bc48cea7bb0fb1a8febf35
Oct 30 19:58:29.328625 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.328583    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:58:29.328709 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.328681    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:58:29.328760 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.328726    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:58:29.342673 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.342636    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:29.439297 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.438389    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l" event={"ID":"57e688d9-46c5-47a1-8150-de9e8e024c49","Type":"ContainerStarted","Data":"ee2d2aa112d61de791a8e6e250487c0daf1922773f771b8bbda09cd3f7751e92"}
Oct 30 19:58:29.439297 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.438442    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l" event={"ID":"57e688d9-46c5-47a1-8150-de9e8e024c49","Type":"ContainerStarted","Data":"212d43f70138f796c317c87ee263c13b9b672d9b91bc48cea7bb0fb1a8febf35"}
Oct 30 19:58:29.441523 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.440979    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9" event={"ID":"505b8b25-660e-432f-87b0-f359078e9f2a","Type":"ContainerStarted","Data":"e715322eac72ff1c96c639b90f4fb91fe32c3e0a14647cd05480cdcf30e7ab1a"}
Oct 30 19:58:29.441523 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.441024    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9" event={"ID":"505b8b25-660e-432f-87b0-f359078e9f2a","Type":"ContainerStarted","Data":"c8120bd38425fdeea9f04953e1e7d726e17a77b0882ea4c8becd993d2c4fdb22"}
Oct 30 19:58:29.482086 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.480676    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l" podStartSLOduration=1.480656232 podStartE2EDuration="1.480656232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:29.469456894 +0000 UTC m=+1588.385063772" watchObservedRunningTime="2026-10-30 19:58:29.480656232 +0000 UTC m=+1588.396263089"
Oct 30 19:58:29.509281 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.509232    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9" podStartSLOduration=1.509218497 podStartE2EDuration="1.509218497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:29.50614434 +0000 UTC m=+1588.421751201" watchObservedRunningTime="2026-10-30 19:58:29.509218497 +0000 UTC m=+1588.424825352"
Oct 30 19:58:29.702480 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.702444    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"]
Oct 30 19:58:29.712314 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.712278    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:58:29.712447 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.712364    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:58:29.712447 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.712408    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 19:58:29.862662 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.862630    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:29.910201 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:29.910170    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:30.457147 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:30.457108    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl" event={"ID":"039cd584-b9e1-4b28-b0d5-5c482cbdaad3","Type":"ContainerStarted","Data":"a046b9f002e980c6e654ea3b39fcc95a3e02a1439f48988672ff9bba5879dfbd"}
Oct 30 19:58:30.457430 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:30.457397    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl" event={"ID":"039cd584-b9e1-4b28-b0d5-5c482cbdaad3","Type":"ContainerStarted","Data":"1375c84e0a4078e436602b76a30dd7c35d3f344ea5828a0097ae3fade5ebf43b"}
Oct 30 19:58:30.535072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:30.534128    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl" podStartSLOduration=2.534087326 podStartE2EDuration="2.534087326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:30.492327635 +0000 UTC m=+1589.407934504" watchObservedRunningTime="2026-10-30 19:58:30.534087326 +0000 UTC m=+1589.449694182"
Oct 30 19:58:30.864589 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:30.864513    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:58:30.865087 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:30.865056    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9" podUID="505b8b25-660e-432f-87b0-f359078e9f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 19:58:30.911809 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:30.911770    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:58:30.912101 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:30.912069    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l" podUID="57e688d9-46c5-47a1-8150-de9e8e024c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 19:58:31.411258 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:31.411141    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:31.413020 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:31.412691    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:58:31.413020 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:31.412743    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl" podUID="039cd584-b9e1-4b28-b0d5-5c482cbdaad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 19:58:31.867351 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:31.867268    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:58:31.867963 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:31.867931    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9" podUID="505b8b25-660e-432f-87b0-f359078e9f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 19:58:31.916237 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:31.916108    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:58:31.916525 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:31.916488    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l" podUID="57e688d9-46c5-47a1-8150-de9e8e024c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 19:58:32.351276 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:32.351037    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:58:32.351492 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:32.351299    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl" podUID="039cd584-b9e1-4b28-b0d5-5c482cbdaad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 19:58:32.862776 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:32.862736    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:58:32.864165 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:32.864101    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9" podUID="505b8b25-660e-432f-87b0-f359078e9f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 19:58:32.931073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:32.914846    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:58:32.931073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:32.914932    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l" podUID="57e688d9-46c5-47a1-8150-de9e8e024c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 19:58:33.389206 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:33.389172    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:58:33.445079 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:33.414365    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl" podUID="039cd584-b9e1-4b28-b0d5-5c482cbdaad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 19:58:33.877128 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:33.873631    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:58:33.877128 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:33.873683    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9" podUID="505b8b25-660e-432f-87b0-f359078e9f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 19:58:33.965072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:33.926261    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:58:33.965072 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:33.926335    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l" podUID="57e688d9-46c5-47a1-8150-de9e8e024c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 19:58:34.359150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:34.344543    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:58:34.359150 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:34.344601    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl" podUID="039cd584-b9e1-4b28-b0d5-5c482cbdaad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 19:58:34.405073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:34.392313    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr" podUID="5983703e-9048-445d-b9db-82e8bcff02a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.141:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:34.657151 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:58:34.648403    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:58:34.711093 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:34.699480    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:58:34.864564 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:34.864488    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:58:34.864776 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:34.864559    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9" podUID="505b8b25-660e-432f-87b0-f359078e9f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 19:58:34.910956 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:34.910872    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:58:34.919336 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:34.919299    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l" podUID="57e688d9-46c5-47a1-8150-de9e8e024c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 19:58:34.975070 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:34.969685    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:35.162922 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.162821    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:35.168152 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.168112    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:35.178170 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.178132    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:35.178786 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.178744    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:58:35.214276 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.214237    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 19:58:35.280073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.276929    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r" podUID="3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:35.312068 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.311230    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6" podUID="0d51a759-22ec-4891-b58e-c8b72e85fbcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": context deadline exceeded"
Oct 30 19:58:35.337166 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.337122    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5" podUID="2db52234-3bbf-4a48-b905-dec230b1b3f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 19:58:35.344261 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.344231    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:58:35.344496 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.344469    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl" podUID="039cd584-b9e1-4b28-b0d5-5c482cbdaad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 19:58:35.397179 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.397135    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr" podUID="5983703e-9048-445d-b9db-82e8bcff02a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.141:8000/ready\": context deadline exceeded"
Oct 30 19:58:35.397940 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.397912    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:35.739432 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.739377    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:35.740719 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.740678    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:58:35.850016 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.849975    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:35.877094 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.866575    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:58:35.877094 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.866686    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9" podUID="505b8b25-660e-432f-87b0-f359078e9f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 19:58:35.921618 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.912527    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725" podUID="864c1436-736d-4921-b137-ed7d41914a74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.139:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:35.921618 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.912660    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:35.921618 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.920397    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:58:35.921618 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.920447    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l" podUID="57e688d9-46c5-47a1-8150-de9e8e024c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 19:58:35.975190 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.975136    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:35.981896 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:35.981867    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:58:36.026136 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.025553    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 19:58:36.172969 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.172918    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:36.173638 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.173611    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:58:36.297147 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.293135    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r" podUID="3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 19:58:36.297147 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.297146    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:58:36.337750 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.337704    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5" podUID="2db52234-3bbf-4a48-b905-dec230b1b3f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:36.363381 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.362698    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:58:36.363643 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.363615    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl" podUID="039cd584-b9e1-4b28-b0d5-5c482cbdaad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 19:58:36.364158 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.364135    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:58:36.404071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.403215    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr" podUID="5983703e-9048-445d-b9db-82e8bcff02a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.141:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:36.667541 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.667510    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 19:58:36.762023 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.761640    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 19:58:36.889852 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.883182    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:58:36.889852 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.883259    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9" podUID="505b8b25-660e-432f-87b0-f359078e9f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 19:58:36.919297 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.919157    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:36.920291 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.920267    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:58:36.936088 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.922525    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725" podUID="864c1436-736d-4921-b137-ed7d41914a74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.139:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:36.993275 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.993227    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:37.025071 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:36.999238    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:58:37.065093 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:37.054196    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:58:37.065093 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:37.054277    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l" podUID="57e688d9-46c5-47a1-8150-de9e8e024c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 19:58:37.074103 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:37.069411    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 19:58:37.087988 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:37.087948    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 19:58:37.127402 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:37.125699    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 19:58:37.168621 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:37.167642    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 19:58:37.191158 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:37.189196    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:37.239079 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:37.238467    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 19:58:37.347153 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:37.347104    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:58:37.347368 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:37.347163    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl" podUID="039cd584-b9e1-4b28-b0d5-5c482cbdaad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 19:58:37.863083 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:37.863031    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:58:37.863364 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:37.863334    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9" podUID="505b8b25-660e-432f-87b0-f359078e9f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 19:58:37.911387 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:37.911359    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:58:37.911629 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:37.911603    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l" podUID="57e688d9-46c5-47a1-8150-de9e8e024c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 19:58:38.368066 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.367460    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:38.375695 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.368985    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:38.375695 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.369846    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 19:58:38.686176 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.686143    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"]
Oct 30 19:58:38.691649 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.691622    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.708986 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.708949    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-981ee5\"/\"openshift-service-ca.crt\""
Oct 30 19:58:38.709189 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.709035    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-981ee5\"/\"kube-root-ca.crt\""
Oct 30 19:58:38.714430 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.714403    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"]
Oct 30 19:58:38.755519 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.755487    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-981ee5\"/\"auth-test-model-981ee5-kserve-self-signed-certs\""
Oct 30 19:58:38.757729 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.757028    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-981ee5\"/\"default-dockercfg-lxpcp\""
Oct 30 19:58:38.793519 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.793483    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-model-cache\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.793817 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.793793    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955ca10f-69cc-4cef-9c87-52beabd36dfa-tls-certs\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.794936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.794335    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-dshm\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.794936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.794600    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-home\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.794936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.794647    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-tmp-dir\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.794936 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.794672    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvvql\" (UniqueName: \"kubernetes.io/projected/955ca10f-69cc-4cef-9c87-52beabd36dfa-kube-api-access-qvvql\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.901069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.896247    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-home\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.901069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.896318    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-tmp-dir\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.901069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.896347    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvql\" (UniqueName: \"kubernetes.io/projected/955ca10f-69cc-4cef-9c87-52beabd36dfa-kube-api-access-qvvql\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.901069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.896497    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-model-cache\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.901069 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.896568    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955ca10f-69cc-4cef-9c87-52beabd36dfa-tls-certs\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.935182 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.932534    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-tmp-dir\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.943423 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.935711    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-model-cache\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.943423 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.896652    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-dshm\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.943423 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.937185    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-home\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.956567 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.956487    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-dshm\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.957572 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.957508    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvvql\" (UniqueName: \"kubernetes.io/projected/955ca10f-69cc-4cef-9c87-52beabd36dfa-kube-api-access-qvvql\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:38.961073 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:38.959671    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955ca10f-69cc-4cef-9c87-52beabd36dfa-tls-certs\") pod \"auth-test-model-981ee5-kserve-6655d955b9-5gcnl\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") " pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:39.018088 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:39.010323    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:39.426441 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:39.426406    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"]
Oct 30 19:58:39.911309 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:39.911271    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:58:39.911629 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:39.911584    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9" podUID="505b8b25-660e-432f-87b0-f359078e9f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:39.933950 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:39.933914    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:58:39.934116 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:39.933971    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l" podUID="57e688d9-46c5-47a1-8150-de9e8e024c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:40.163458 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:40.163370    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl" event={"ID":"955ca10f-69cc-4cef-9c87-52beabd36dfa","Type":"ContainerStarted","Data":"3faba949020f4462db0ece1bd9564b2d0dbf61891f4807651b42ddc387fa45d2"}
Oct 30 19:58:40.163458 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:40.163419    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl" event={"ID":"955ca10f-69cc-4cef-9c87-52beabd36dfa","Type":"ContainerStarted","Data":"d7efc7a21f885c6f80643177ee5a635d8c238194ce65c134387da0c0556674c7"}
Oct 30 19:58:40.193902 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:40.192688    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl" podStartSLOduration=2.19266965 podStartE2EDuration="2.19266965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:40.190096283 +0000 UTC m=+1599.105703160" watchObservedRunningTime="2026-10-30 19:58:40.19266965 +0000 UTC m=+1599.108276508"
Oct 30 19:58:40.408459 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:40.408418    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:40.409469 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:40.409434    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:40.418435 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:40.418289    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 19:58:40.934348 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:40.934316    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:58:40.934744 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:40.934371    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l" podUID="57e688d9-46c5-47a1-8150-de9e8e024c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:40.950959 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:40.950922    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:40.951469 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:40.951448    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:40.952473 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:40.952454    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 19:58:41.011019 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:41.010983    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:49.011472 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:49.011439    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:49.023945 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:49.023922    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:58:49.234804 ip-10-0-142-167 kubenswrapper[6030]: I1030 19:58:49.234776    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 19:59:04.288774 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:59:04.288738    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 19:59:34.291444 ip-10-0-142-167 kubenswrapper[6030]: E1030 19:59:34.291406    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:00:00.172430 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.172398    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf"]
Oct 30 20:00:00.176703 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.176686    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf"
Oct 30 20:00:00.203609 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.203579    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h"]
Oct 30 20:00:00.209535 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.209508    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h"
Oct 30 20:00:00.209972 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.209955    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf"]
Oct 30 20:00:00.214401 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.214372    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px"]
Oct 30 20:00:00.219225 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.219199    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs"]
Oct 30 20:00:00.221129 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.221107    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px"
Oct 30 20:00:00.225570 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.225538    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs"
Oct 30 20:00:00.252643 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.252600    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h"]
Oct 30 20:00:00.259476 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.259448    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl66f\" (UniqueName: \"kubernetes.io/projected/bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78-kube-api-access-hl66f\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf\" (UID: \"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf"
Oct 30 20:00:00.259805 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.259647    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf\" (UID: \"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf"
Oct 30 20:00:00.260733 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.260712    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww"]
Oct 30 20:00:00.268238 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.268212    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px"]
Oct 30 20:00:00.268393 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.268343    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww"
Oct 30 20:00:00.278928 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.278614    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 20:00:00.282018 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.281997    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr"]
Oct 30 20:00:00.294943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.291285    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww"]
Oct 30 20:00:00.294943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.291396    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr"
Oct 30 20:00:00.294943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.293959    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs"]
Oct 30 20:00:00.312684 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.310654    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr"]
Oct 30 20:00:00.361066 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.361019    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06fcfd9-153b-402a-bce4-c64373fcdb60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs\" (UID: \"f06fcfd9-153b-402a-bce4-c64373fcdb60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs"
Oct 30 20:00:00.361262 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.361097    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/224aad14-bda8-4fbc-a79b-ea90309b5227-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px\" (UID: \"224aad14-bda8-4fbc-a79b-ea90309b5227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px"
Oct 30 20:00:00.361320 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.361268    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl66f\" (UniqueName: \"kubernetes.io/projected/bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78-kube-api-access-hl66f\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf\" (UID: \"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf"
Oct 30 20:00:00.361381 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.361313    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct2s7\" (UniqueName: \"kubernetes.io/projected/e58ffa2d-b351-4239-9921-26d669955e92-kube-api-access-ct2s7\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h\" (UID: \"e58ffa2d-b351-4239-9921-26d669955e92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h"
Oct 30 20:00:00.361438 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.361396    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvwt4\" (UniqueName: \"kubernetes.io/projected/2cc80851-9204-48e4-a739-3d7399e6449a-kube-api-access-cvwt4\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr\" (UID: \"2cc80851-9204-48e4-a739-3d7399e6449a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr"
Oct 30 20:00:00.361607 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.361586    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/267c8f37-616e-4b1e-91ee-804399ee4b9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww\" (UID: \"267c8f37-616e-4b1e-91ee-804399ee4b9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww"
Oct 30 20:00:00.361663 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.361628    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw62x\" (UniqueName: \"kubernetes.io/projected/f06fcfd9-153b-402a-bce4-c64373fcdb60-kube-api-access-hw62x\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs\" (UID: \"f06fcfd9-153b-402a-bce4-c64373fcdb60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs"
Oct 30 20:00:00.361718 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.361658    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zb6z\" (UniqueName: \"kubernetes.io/projected/224aad14-bda8-4fbc-a79b-ea90309b5227-kube-api-access-2zb6z\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px\" (UID: \"224aad14-bda8-4fbc-a79b-ea90309b5227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px"
Oct 30 20:00:00.361782 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.361751    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58ffa2d-b351-4239-9921-26d669955e92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h\" (UID: \"e58ffa2d-b351-4239-9921-26d669955e92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h"
Oct 30 20:00:00.361853 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.361797    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf\" (UID: \"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf"
Oct 30 20:00:00.361910 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.361847    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc80851-9204-48e4-a739-3d7399e6449a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr\" (UID: \"2cc80851-9204-48e4-a739-3d7399e6449a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr"
Oct 30 20:00:00.361910 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.361876    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9tsj\" (UniqueName: \"kubernetes.io/projected/267c8f37-616e-4b1e-91ee-804399ee4b9f-kube-api-access-n9tsj\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww\" (UID: \"267c8f37-616e-4b1e-91ee-804399ee4b9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww"
Oct 30 20:00:00.362372 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.362355    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf\" (UID: \"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf"
Oct 30 20:00:00.377703 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.377677    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl66f\" (UniqueName: \"kubernetes.io/projected/bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78-kube-api-access-hl66f\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf\" (UID: \"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf"
Oct 30 20:00:00.463084 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.462975    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct2s7\" (UniqueName: \"kubernetes.io/projected/e58ffa2d-b351-4239-9921-26d669955e92-kube-api-access-ct2s7\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h\" (UID: \"e58ffa2d-b351-4239-9921-26d669955e92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h"
Oct 30 20:00:00.463084 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.463034    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvwt4\" (UniqueName: \"kubernetes.io/projected/2cc80851-9204-48e4-a739-3d7399e6449a-kube-api-access-cvwt4\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr\" (UID: \"2cc80851-9204-48e4-a739-3d7399e6449a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr"
Oct 30 20:00:00.463322 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.463133    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/267c8f37-616e-4b1e-91ee-804399ee4b9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww\" (UID: \"267c8f37-616e-4b1e-91ee-804399ee4b9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww"
Oct 30 20:00:00.463322 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.463160    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw62x\" (UniqueName: \"kubernetes.io/projected/f06fcfd9-153b-402a-bce4-c64373fcdb60-kube-api-access-hw62x\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs\" (UID: \"f06fcfd9-153b-402a-bce4-c64373fcdb60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs"
Oct 30 20:00:00.463322 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.463180    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zb6z\" (UniqueName: \"kubernetes.io/projected/224aad14-bda8-4fbc-a79b-ea90309b5227-kube-api-access-2zb6z\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px\" (UID: \"224aad14-bda8-4fbc-a79b-ea90309b5227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px"
Oct 30 20:00:00.463322 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.463201    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58ffa2d-b351-4239-9921-26d669955e92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h\" (UID: \"e58ffa2d-b351-4239-9921-26d669955e92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h"
Oct 30 20:00:00.463322 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.463227    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc80851-9204-48e4-a739-3d7399e6449a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr\" (UID: \"2cc80851-9204-48e4-a739-3d7399e6449a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr"
Oct 30 20:00:00.463322 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.463250    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9tsj\" (UniqueName: \"kubernetes.io/projected/267c8f37-616e-4b1e-91ee-804399ee4b9f-kube-api-access-n9tsj\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww\" (UID: \"267c8f37-616e-4b1e-91ee-804399ee4b9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww"
Oct 30 20:00:00.463322 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.463321    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06fcfd9-153b-402a-bce4-c64373fcdb60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs\" (UID: \"f06fcfd9-153b-402a-bce4-c64373fcdb60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs"
Oct 30 20:00:00.463708 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.463655    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/224aad14-bda8-4fbc-a79b-ea90309b5227-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px\" (UID: \"224aad14-bda8-4fbc-a79b-ea90309b5227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px"
Oct 30 20:00:00.463993 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.463977    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc80851-9204-48e4-a739-3d7399e6449a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr\" (UID: \"2cc80851-9204-48e4-a739-3d7399e6449a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr"
Oct 30 20:00:00.464080 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.464037    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58ffa2d-b351-4239-9921-26d669955e92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h\" (UID: \"e58ffa2d-b351-4239-9921-26d669955e92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h"
Oct 30 20:00:00.464128 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.464063    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/267c8f37-616e-4b1e-91ee-804399ee4b9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww\" (UID: \"267c8f37-616e-4b1e-91ee-804399ee4b9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww"
Oct 30 20:00:00.464128 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.464101    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06fcfd9-153b-402a-bce4-c64373fcdb60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs\" (UID: \"f06fcfd9-153b-402a-bce4-c64373fcdb60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs"
Oct 30 20:00:00.464335 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.464317    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/224aad14-bda8-4fbc-a79b-ea90309b5227-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px\" (UID: \"224aad14-bda8-4fbc-a79b-ea90309b5227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px"
Oct 30 20:00:00.478256 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.478225    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9tsj\" (UniqueName: \"kubernetes.io/projected/267c8f37-616e-4b1e-91ee-804399ee4b9f-kube-api-access-n9tsj\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww\" (UID: \"267c8f37-616e-4b1e-91ee-804399ee4b9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww"
Oct 30 20:00:00.478256 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.478251    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zb6z\" (UniqueName: \"kubernetes.io/projected/224aad14-bda8-4fbc-a79b-ea90309b5227-kube-api-access-2zb6z\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px\" (UID: \"224aad14-bda8-4fbc-a79b-ea90309b5227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px"
Oct 30 20:00:00.478462 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.478275    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvwt4\" (UniqueName: \"kubernetes.io/projected/2cc80851-9204-48e4-a739-3d7399e6449a-kube-api-access-cvwt4\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr\" (UID: \"2cc80851-9204-48e4-a739-3d7399e6449a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr"
Oct 30 20:00:00.478674 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.478651    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct2s7\" (UniqueName: \"kubernetes.io/projected/e58ffa2d-b351-4239-9921-26d669955e92-kube-api-access-ct2s7\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h\" (UID: \"e58ffa2d-b351-4239-9921-26d669955e92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h"
Oct 30 20:00:00.483202 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.483179    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw62x\" (UniqueName: \"kubernetes.io/projected/f06fcfd9-153b-402a-bce4-c64373fcdb60-kube-api-access-hw62x\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs\" (UID: \"f06fcfd9-153b-402a-bce4-c64373fcdb60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs"
Oct 30 20:00:00.493889 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.493871    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf"
Oct 30 20:00:00.551032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.548924    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h"
Oct 30 20:00:00.568636 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.566748    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px"
Oct 30 20:00:00.579164 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.578891    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs"
Oct 30 20:00:00.603398 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.603346    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww"
Oct 30 20:00:00.633013 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.619949    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr"
Oct 30 20:00:00.796833 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.796797    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf"]
Oct 30 20:00:00.806394 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:00:00.806360    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc99f4e_ef07_48b6_a8fd_f01fc99b3f78.slice/crio-04c336e66bcd559c1d6782df22950452fcc4c96cb5138c578202e0283292b23e WatchSource:0}: Error finding container 04c336e66bcd559c1d6782df22950452fcc4c96cb5138c578202e0283292b23e: Status 404 returned error can't find the container with id 04c336e66bcd559c1d6782df22950452fcc4c96cb5138c578202e0283292b23e
Oct 30 20:00:00.892302 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.892269    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px"]
Oct 30 20:00:00.894692 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:00:00.894649    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58ffa2d_b351_4239_9921_26d669955e92.slice/crio-e424d00cdbede5e518dc46db312804bf363f800993f695a8a68e08503e2f04c0 WatchSource:0}: Error finding container e424d00cdbede5e518dc46db312804bf363f800993f695a8a68e08503e2f04c0: Status 404 returned error can't find the container with id e424d00cdbede5e518dc46db312804bf363f800993f695a8a68e08503e2f04c0
Oct 30 20:00:00.895373 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:00:00.895329    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224aad14_bda8_4fbc_a79b_ea90309b5227.slice/crio-8e9fe096356687724d524e70394eab93a58e50b4d5c9de46b8631731a5417164 WatchSource:0}: Error finding container 8e9fe096356687724d524e70394eab93a58e50b4d5c9de46b8631731a5417164: Status 404 returned error can't find the container with id 8e9fe096356687724d524e70394eab93a58e50b4d5c9de46b8631731a5417164
Oct 30 20:00:00.896963 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.896939    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h"]
Oct 30 20:00:00.935694 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.935662    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs"]
Oct 30 20:00:00.961553 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.961518    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h" event={"ID":"e58ffa2d-b351-4239-9921-26d669955e92","Type":"ContainerStarted","Data":"e424d00cdbede5e518dc46db312804bf363f800993f695a8a68e08503e2f04c0"}
Oct 30 20:00:00.962973 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.962950    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs" event={"ID":"f06fcfd9-153b-402a-bce4-c64373fcdb60","Type":"ContainerStarted","Data":"096fb54078ac5ddbfb5e35a81c6207ce351d17d1d20216dbaf94716896e87f7a"}
Oct 30 20:00:00.964358 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.964335    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf" event={"ID":"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78","Type":"ContainerStarted","Data":"04c336e66bcd559c1d6782df22950452fcc4c96cb5138c578202e0283292b23e"}
Oct 30 20:00:00.966536 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:00.966515    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px" event={"ID":"224aad14-bda8-4fbc-a79b-ea90309b5227","Type":"ContainerStarted","Data":"8e9fe096356687724d524e70394eab93a58e50b4d5c9de46b8631731a5417164"}
Oct 30 20:00:01.013790 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:01.013766    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww"]
Oct 30 20:00:01.060455 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:01.060434    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr"]
Oct 30 20:00:01.072327 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:00:01.072287    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267c8f37_616e_4b1e_91ee_804399ee4b9f.slice/crio-41ddb7e1e83fc9a807cea6e5695195c9ee36f0c0e01b5ba8baaa3d150897a8ab WatchSource:0}: Error finding container 41ddb7e1e83fc9a807cea6e5695195c9ee36f0c0e01b5ba8baaa3d150897a8ab: Status 404 returned error can't find the container with id 41ddb7e1e83fc9a807cea6e5695195c9ee36f0c0e01b5ba8baaa3d150897a8ab
Oct 30 20:00:01.073210 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:00:01.073189    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc80851_9204_48e4_a739_3d7399e6449a.slice/crio-b537d0904ee88907381242f83eb4c14672bed1e97564078e09f8f08ebba8cb79 WatchSource:0}: Error finding container b537d0904ee88907381242f83eb4c14672bed1e97564078e09f8f08ebba8cb79: Status 404 returned error can't find the container with id b537d0904ee88907381242f83eb4c14672bed1e97564078e09f8f08ebba8cb79
Oct 30 20:00:01.978110 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:01.978075    6030 generic.go:356] "Generic (PLEG): container finished" podID="224aad14-bda8-4fbc-a79b-ea90309b5227" containerID="47b5dd8c29edbce72d7fa312bedcd084134595d3db7627e5656645c7fdfd0654" exitCode=60
Oct 30 20:00:01.978506 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:01.978244    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px" event={"ID":"224aad14-bda8-4fbc-a79b-ea90309b5227","Type":"ContainerDied","Data":"47b5dd8c29edbce72d7fa312bedcd084134595d3db7627e5656645c7fdfd0654"}
Oct 30 20:00:01.978575 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:01.978545    6030 scope.go:122] "RemoveContainer" containerID="47b5dd8c29edbce72d7fa312bedcd084134595d3db7627e5656645c7fdfd0654"
Oct 30 20:00:01.980893 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:01.980872    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww" event={"ID":"267c8f37-616e-4b1e-91ee-804399ee4b9f","Type":"ContainerStarted","Data":"b650829bb60b4faec91e14beee943d0749cb94e11a4b034473bd8b93f6a743d7"}
Oct 30 20:00:01.980971 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:01.980913    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww" event={"ID":"267c8f37-616e-4b1e-91ee-804399ee4b9f","Type":"ContainerStarted","Data":"41ddb7e1e83fc9a807cea6e5695195c9ee36f0c0e01b5ba8baaa3d150897a8ab"}
Oct 30 20:00:01.983313 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:01.982787    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h" event={"ID":"e58ffa2d-b351-4239-9921-26d669955e92","Type":"ContainerStarted","Data":"65fcda7af7172876bad3a8c7daf731318d38e5f35440f9352322f7cc61907b05"}
Oct 30 20:00:01.985421 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:01.985394    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs" event={"ID":"f06fcfd9-153b-402a-bce4-c64373fcdb60","Type":"ContainerStarted","Data":"72a8e8278dd9d0dd1e20154416919f688c7b70e7100fbc2ffece48f7453027ef"}
Oct 30 20:00:01.987103 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:01.987079    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr" event={"ID":"2cc80851-9204-48e4-a739-3d7399e6449a","Type":"ContainerStarted","Data":"6a439473fb699b8de5cf191460bb13e2f47f551da2d41ef43610297eb4c5ee44"}
Oct 30 20:00:01.987224 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:01.987140    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr" event={"ID":"2cc80851-9204-48e4-a739-3d7399e6449a","Type":"ContainerStarted","Data":"b537d0904ee88907381242f83eb4c14672bed1e97564078e09f8f08ebba8cb79"}
Oct 30 20:00:01.989317 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:01.989245    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf" event={"ID":"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78","Type":"ContainerStarted","Data":"d52dcdf92767b2f02f7a45fe9e2facd86f7c092231a43227740b09d9d507bc5c"}
Oct 30 20:00:02.026679 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:02.026599    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs" podStartSLOduration=2.026575381 podStartE2EDuration="2.026575381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.024294441 +0000 UTC m=+1680.939901297" watchObservedRunningTime="2026-10-30 20:00:02.026575381 +0000 UTC m=+1680.942182231"
Oct 30 20:00:02.053612 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:02.053548    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf" podStartSLOduration=2.053535282 podStartE2EDuration="2.053535282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.049693727 +0000 UTC m=+1680.965300584" watchObservedRunningTime="2026-10-30 20:00:02.053535282 +0000 UTC m=+1680.969142138"
Oct 30 20:00:02.078529 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:02.078486    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww" podStartSLOduration=2.078470466 podStartE2EDuration="2.078470466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.074421533 +0000 UTC m=+1680.990028391" watchObservedRunningTime="2026-10-30 20:00:02.078470466 +0000 UTC m=+1680.994077325"
Oct 30 20:00:02.106418 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:02.106376    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h" podStartSLOduration=2.10636289 podStartE2EDuration="2.10636289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.103456418 +0000 UTC m=+1681.019063278" watchObservedRunningTime="2026-10-30 20:00:02.10636289 +0000 UTC m=+1681.021969746"
Oct 30 20:00:02.127467 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:02.127370    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr" podStartSLOduration=2.127342822 podStartE2EDuration="2.127342822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.122610321 +0000 UTC m=+1681.038217179" watchObservedRunningTime="2026-10-30 20:00:02.127342822 +0000 UTC m=+1681.042949671"
Oct 30 20:00:02.998384 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:02.998349    6030 generic.go:356] "Generic (PLEG): container finished" podID="2cc80851-9204-48e4-a739-3d7399e6449a" containerID="6a439473fb699b8de5cf191460bb13e2f47f551da2d41ef43610297eb4c5ee44" exitCode=60
Oct 30 20:00:02.998764 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:02.998442    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr" event={"ID":"2cc80851-9204-48e4-a739-3d7399e6449a","Type":"ContainerDied","Data":"6a439473fb699b8de5cf191460bb13e2f47f551da2d41ef43610297eb4c5ee44"}
Oct 30 20:00:02.998832 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:02.998797    6030 scope.go:122] "RemoveContainer" containerID="6a439473fb699b8de5cf191460bb13e2f47f551da2d41ef43610297eb4c5ee44"
Oct 30 20:00:03.000686 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:03.000549    6030 generic.go:356] "Generic (PLEG): container finished" podID="bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78" containerID="d52dcdf92767b2f02f7a45fe9e2facd86f7c092231a43227740b09d9d507bc5c" exitCode=60
Oct 30 20:00:03.000686 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:03.000579    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf" event={"ID":"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78","Type":"ContainerDied","Data":"d52dcdf92767b2f02f7a45fe9e2facd86f7c092231a43227740b09d9d507bc5c"}
Oct 30 20:00:03.000893 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:03.000872    6030 scope.go:122] "RemoveContainer" containerID="d52dcdf92767b2f02f7a45fe9e2facd86f7c092231a43227740b09d9d507bc5c"
Oct 30 20:00:03.003001 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:03.002980    6030 generic.go:356] "Generic (PLEG): container finished" podID="224aad14-bda8-4fbc-a79b-ea90309b5227" containerID="4d15f91016098fe2aa32382c1c5645f3798b32f72628bf58b80dc2dc941424dd" exitCode=60
Oct 30 20:00:03.003175 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:03.003153    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px" event={"ID":"224aad14-bda8-4fbc-a79b-ea90309b5227","Type":"ContainerDied","Data":"4d15f91016098fe2aa32382c1c5645f3798b32f72628bf58b80dc2dc941424dd"}
Oct 30 20:00:03.003305 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:03.003205    6030 scope.go:122] "RemoveContainer" containerID="47b5dd8c29edbce72d7fa312bedcd084134595d3db7627e5656645c7fdfd0654"
Oct 30 20:00:03.003491 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:03.003443    6030 scope.go:122] "RemoveContainer" containerID="4d15f91016098fe2aa32382c1c5645f3798b32f72628bf58b80dc2dc941424dd"
Oct 30 20:00:03.004034 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:03.004007    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px_odh-ai-gateway-infra(224aad14-bda8-4fbc-a79b-ea90309b5227)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px" podUID="224aad14-bda8-4fbc-a79b-ea90309b5227"
Oct 30 20:00:03.005214 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:03.005188    6030 generic.go:356] "Generic (PLEG): container finished" podID="267c8f37-616e-4b1e-91ee-804399ee4b9f" containerID="b650829bb60b4faec91e14beee943d0749cb94e11a4b034473bd8b93f6a743d7" exitCode=60
Oct 30 20:00:03.005333 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:03.005222    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww" event={"ID":"267c8f37-616e-4b1e-91ee-804399ee4b9f","Type":"ContainerDied","Data":"b650829bb60b4faec91e14beee943d0749cb94e11a4b034473bd8b93f6a743d7"}
Oct 30 20:00:03.005494 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:03.005477    6030 scope.go:122] "RemoveContainer" containerID="b650829bb60b4faec91e14beee943d0749cb94e11a4b034473bd8b93f6a743d7"
Oct 30 20:00:03.007247 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:03.007226    6030 generic.go:356] "Generic (PLEG): container finished" podID="e58ffa2d-b351-4239-9921-26d669955e92" containerID="65fcda7af7172876bad3a8c7daf731318d38e5f35440f9352322f7cc61907b05" exitCode=60
Oct 30 20:00:03.007348 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:03.007309    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h" event={"ID":"e58ffa2d-b351-4239-9921-26d669955e92","Type":"ContainerDied","Data":"65fcda7af7172876bad3a8c7daf731318d38e5f35440f9352322f7cc61907b05"}
Oct 30 20:00:03.007477 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:03.007464    6030 scope.go:122] "RemoveContainer" containerID="65fcda7af7172876bad3a8c7daf731318d38e5f35440f9352322f7cc61907b05"
Oct 30 20:00:03.008784 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:03.008765    6030 generic.go:356] "Generic (PLEG): container finished" podID="f06fcfd9-153b-402a-bce4-c64373fcdb60" containerID="72a8e8278dd9d0dd1e20154416919f688c7b70e7100fbc2ffece48f7453027ef" exitCode=60
Oct 30 20:00:03.008879 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:03.008801    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs" event={"ID":"f06fcfd9-153b-402a-bce4-c64373fcdb60","Type":"ContainerDied","Data":"72a8e8278dd9d0dd1e20154416919f688c7b70e7100fbc2ffece48f7453027ef"}
Oct 30 20:00:03.009037 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:03.009020    6030 scope.go:122] "RemoveContainer" containerID="72a8e8278dd9d0dd1e20154416919f688c7b70e7100fbc2ffece48f7453027ef"
Oct 30 20:00:04.018343 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:04.018311    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs" event={"ID":"f06fcfd9-153b-402a-bce4-c64373fcdb60","Type":"ContainerStarted","Data":"b02f5f87e0bae542c7027581a38ac72c901bf7d498ed0c2c63087d5cb7a19a73"}
Oct 30 20:00:04.020436 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:04.020402    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr" event={"ID":"2cc80851-9204-48e4-a739-3d7399e6449a","Type":"ContainerStarted","Data":"cba4b71d7cb723df8d14b7f51d91b5f051202eef0f1a49b6ce7c3b807e54aca7"}
Oct 30 20:00:04.022365 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:04.022338    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf" event={"ID":"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78","Type":"ContainerStarted","Data":"e4c288c6d0cd2827fddb6bb40f2fdfb65053aeaf108d16d5eba86dd91090b7a2"}
Oct 30 20:00:04.024133 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:04.024111    6030 scope.go:122] "RemoveContainer" containerID="4d15f91016098fe2aa32382c1c5645f3798b32f72628bf58b80dc2dc941424dd"
Oct 30 20:00:04.024445 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:04.024419    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px_odh-ai-gateway-infra(224aad14-bda8-4fbc-a79b-ea90309b5227)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px" podUID="224aad14-bda8-4fbc-a79b-ea90309b5227"
Oct 30 20:00:04.026101 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:04.026019    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww" event={"ID":"267c8f37-616e-4b1e-91ee-804399ee4b9f","Type":"ContainerStarted","Data":"2d005189b075b35bec290d3f998ba719484ec3e83cbb883511ab1209e3b0f1b4"}
Oct 30 20:00:04.028036 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:04.028016    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h" event={"ID":"e58ffa2d-b351-4239-9921-26d669955e92","Type":"ContainerStarted","Data":"7db9de846788895eb46b627f998aa6bdce49a27a31a851360b6cef10910f2da0"}
Oct 30 20:00:04.290409 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:04.290326    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:00:05.036432 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.036397    6030 generic.go:356] "Generic (PLEG): container finished" podID="2cc80851-9204-48e4-a739-3d7399e6449a" containerID="cba4b71d7cb723df8d14b7f51d91b5f051202eef0f1a49b6ce7c3b807e54aca7" exitCode=60
Oct 30 20:00:05.036953 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.036498    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr" event={"ID":"2cc80851-9204-48e4-a739-3d7399e6449a","Type":"ContainerDied","Data":"cba4b71d7cb723df8d14b7f51d91b5f051202eef0f1a49b6ce7c3b807e54aca7"}
Oct 30 20:00:05.036953 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.036567    6030 scope.go:122] "RemoveContainer" containerID="6a439473fb699b8de5cf191460bb13e2f47f551da2d41ef43610297eb4c5ee44"
Oct 30 20:00:05.036953 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.036825    6030 scope.go:122] "RemoveContainer" containerID="cba4b71d7cb723df8d14b7f51d91b5f051202eef0f1a49b6ce7c3b807e54aca7"
Oct 30 20:00:05.037256 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:05.037230    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr_odh-ai-gateway-infra(2cc80851-9204-48e4-a739-3d7399e6449a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr" podUID="2cc80851-9204-48e4-a739-3d7399e6449a"
Oct 30 20:00:05.039317 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.039290    6030 generic.go:356] "Generic (PLEG): container finished" podID="bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78" containerID="e4c288c6d0cd2827fddb6bb40f2fdfb65053aeaf108d16d5eba86dd91090b7a2" exitCode=60
Oct 30 20:00:05.039457 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.039375    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf" event={"ID":"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78","Type":"ContainerDied","Data":"e4c288c6d0cd2827fddb6bb40f2fdfb65053aeaf108d16d5eba86dd91090b7a2"}
Oct 30 20:00:05.039673 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.039656    6030 scope.go:122] "RemoveContainer" containerID="e4c288c6d0cd2827fddb6bb40f2fdfb65053aeaf108d16d5eba86dd91090b7a2"
Oct 30 20:00:05.040072 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:05.040025    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf_odh-ai-gateway-infra(bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf" podUID="bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78"
Oct 30 20:00:05.041608 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.041561    6030 generic.go:356] "Generic (PLEG): container finished" podID="267c8f37-616e-4b1e-91ee-804399ee4b9f" containerID="2d005189b075b35bec290d3f998ba719484ec3e83cbb883511ab1209e3b0f1b4" exitCode=60
Oct 30 20:00:05.041731 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.041641    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww" event={"ID":"267c8f37-616e-4b1e-91ee-804399ee4b9f","Type":"ContainerDied","Data":"2d005189b075b35bec290d3f998ba719484ec3e83cbb883511ab1209e3b0f1b4"}
Oct 30 20:00:05.041895 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.041881    6030 scope.go:122] "RemoveContainer" containerID="2d005189b075b35bec290d3f998ba719484ec3e83cbb883511ab1209e3b0f1b4"
Oct 30 20:00:05.042196 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:05.042172    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww_odh-ai-gateway-infra(267c8f37-616e-4b1e-91ee-804399ee4b9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww" podUID="267c8f37-616e-4b1e-91ee-804399ee4b9f"
Oct 30 20:00:05.043815 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.043793    6030 generic.go:356] "Generic (PLEG): container finished" podID="e58ffa2d-b351-4239-9921-26d669955e92" containerID="7db9de846788895eb46b627f998aa6bdce49a27a31a851360b6cef10910f2da0" exitCode=60
Oct 30 20:00:05.043923 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.043876    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h" event={"ID":"e58ffa2d-b351-4239-9921-26d669955e92","Type":"ContainerDied","Data":"7db9de846788895eb46b627f998aa6bdce49a27a31a851360b6cef10910f2da0"}
Oct 30 20:00:05.044138 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.044122    6030 scope.go:122] "RemoveContainer" containerID="7db9de846788895eb46b627f998aa6bdce49a27a31a851360b6cef10910f2da0"
Oct 30 20:00:05.044447 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:05.044427    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h_odh-ai-gateway-infra(e58ffa2d-b351-4239-9921-26d669955e92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h" podUID="e58ffa2d-b351-4239-9921-26d669955e92"
Oct 30 20:00:05.046365 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.046332    6030 generic.go:356] "Generic (PLEG): container finished" podID="f06fcfd9-153b-402a-bce4-c64373fcdb60" containerID="b02f5f87e0bae542c7027581a38ac72c901bf7d498ed0c2c63087d5cb7a19a73" exitCode=60
Oct 30 20:00:05.046468 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.046387    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs" event={"ID":"f06fcfd9-153b-402a-bce4-c64373fcdb60","Type":"ContainerDied","Data":"b02f5f87e0bae542c7027581a38ac72c901bf7d498ed0c2c63087d5cb7a19a73"}
Oct 30 20:00:05.046626 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.046612    6030 scope.go:122] "RemoveContainer" containerID="b02f5f87e0bae542c7027581a38ac72c901bf7d498ed0c2c63087d5cb7a19a73"
Oct 30 20:00:05.046884 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:05.046867    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs_odh-ai-gateway-infra(f06fcfd9-153b-402a-bce4-c64373fcdb60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs" podUID="f06fcfd9-153b-402a-bce4-c64373fcdb60"
Oct 30 20:00:05.059611 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.059593    6030 scope.go:122] "RemoveContainer" containerID="d52dcdf92767b2f02f7a45fe9e2facd86f7c092231a43227740b09d9d507bc5c"
Oct 30 20:00:05.073260 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.073241    6030 scope.go:122] "RemoveContainer" containerID="b650829bb60b4faec91e14beee943d0749cb94e11a4b034473bd8b93f6a743d7"
Oct 30 20:00:05.087988 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.087965    6030 scope.go:122] "RemoveContainer" containerID="65fcda7af7172876bad3a8c7daf731318d38e5f35440f9352322f7cc61907b05"
Oct 30 20:00:05.101796 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:05.101770    6030 scope.go:122] "RemoveContainer" containerID="72a8e8278dd9d0dd1e20154416919f688c7b70e7100fbc2ffece48f7453027ef"
Oct 30 20:00:06.056165 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:06.056136    6030 scope.go:122] "RemoveContainer" containerID="7db9de846788895eb46b627f998aa6bdce49a27a31a851360b6cef10910f2da0"
Oct 30 20:00:06.056706 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:06.056475    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h_odh-ai-gateway-infra(e58ffa2d-b351-4239-9921-26d669955e92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h" podUID="e58ffa2d-b351-4239-9921-26d669955e92"
Oct 30 20:00:06.057727 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:06.057705    6030 scope.go:122] "RemoveContainer" containerID="b02f5f87e0bae542c7027581a38ac72c901bf7d498ed0c2c63087d5cb7a19a73"
Oct 30 20:00:06.058027 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:06.058003    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs_odh-ai-gateway-infra(f06fcfd9-153b-402a-bce4-c64373fcdb60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs" podUID="f06fcfd9-153b-402a-bce4-c64373fcdb60"
Oct 30 20:00:06.059203 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:06.059182    6030 scope.go:122] "RemoveContainer" containerID="cba4b71d7cb723df8d14b7f51d91b5f051202eef0f1a49b6ce7c3b807e54aca7"
Oct 30 20:00:06.059472 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:06.059450    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr_odh-ai-gateway-infra(2cc80851-9204-48e4-a739-3d7399e6449a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr" podUID="2cc80851-9204-48e4-a739-3d7399e6449a"
Oct 30 20:00:06.060627 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:06.060609    6030 scope.go:122] "RemoveContainer" containerID="e4c288c6d0cd2827fddb6bb40f2fdfb65053aeaf108d16d5eba86dd91090b7a2"
Oct 30 20:00:06.060892 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:06.060872    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf_odh-ai-gateway-infra(bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf" podUID="bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78"
Oct 30 20:00:06.062198 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:06.062181    6030 scope.go:122] "RemoveContainer" containerID="2d005189b075b35bec290d3f998ba719484ec3e83cbb883511ab1209e3b0f1b4"
Oct 30 20:00:06.062395 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:06.062378    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww_odh-ai-gateway-infra(267c8f37-616e-4b1e-91ee-804399ee4b9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww" podUID="267c8f37-616e-4b1e-91ee-804399ee4b9f"
Oct 30 20:00:15.203979 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:15.203945    6030 scope.go:122] "RemoveContainer" containerID="2d005189b075b35bec290d3f998ba719484ec3e83cbb883511ab1209e3b0f1b4"
Oct 30 20:00:15.204530 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:15.204330    6030 scope.go:122] "RemoveContainer" containerID="e4c288c6d0cd2827fddb6bb40f2fdfb65053aeaf108d16d5eba86dd91090b7a2"
Oct 30 20:00:16.149350 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:16.149308    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww" event={"ID":"267c8f37-616e-4b1e-91ee-804399ee4b9f","Type":"ContainerStarted","Data":"a3884bb997e155804bf6bfef5ecbeb805050f3f8d35d4fdab8dd263baaf1a57a"}
Oct 30 20:00:16.151477 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:16.151445    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf" event={"ID":"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78","Type":"ContainerStarted","Data":"5a4e143a37f24d2c1c263e186ea341d3bb42999bb1194684cad722c8b2cc1547"}
Oct 30 20:00:16.194080 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:16.194028    6030 scope.go:122] "RemoveContainer" containerID="cba4b71d7cb723df8d14b7f51d91b5f051202eef0f1a49b6ce7c3b807e54aca7"
Oct 30 20:00:16.194208 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:16.194138    6030 scope.go:122] "RemoveContainer" containerID="4d15f91016098fe2aa32382c1c5645f3798b32f72628bf58b80dc2dc941424dd"
Oct 30 20:00:16.194743 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:16.194672    6030 scope.go:122] "RemoveContainer" containerID="b02f5f87e0bae542c7027581a38ac72c901bf7d498ed0c2c63087d5cb7a19a73"
Oct 30 20:00:17.160906 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:17.160859    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr" event={"ID":"2cc80851-9204-48e4-a739-3d7399e6449a","Type":"ContainerStarted","Data":"43c7a0fd61aa151a23559672839d1eaff8e35b146b6755fb6a96f3c68d5efdfe"}
Oct 30 20:00:17.162844 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:17.162820    6030 generic.go:356] "Generic (PLEG): container finished" podID="bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78" containerID="5a4e143a37f24d2c1c263e186ea341d3bb42999bb1194684cad722c8b2cc1547" exitCode=60
Oct 30 20:00:17.162972 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:17.162886    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf" event={"ID":"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78","Type":"ContainerDied","Data":"5a4e143a37f24d2c1c263e186ea341d3bb42999bb1194684cad722c8b2cc1547"}
Oct 30 20:00:17.162972 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:17.162938    6030 scope.go:122] "RemoveContainer" containerID="e4c288c6d0cd2827fddb6bb40f2fdfb65053aeaf108d16d5eba86dd91090b7a2"
Oct 30 20:00:17.163359 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:17.163334    6030 scope.go:122] "RemoveContainer" containerID="5a4e143a37f24d2c1c263e186ea341d3bb42999bb1194684cad722c8b2cc1547"
Oct 30 20:00:17.163694 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:17.163660    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf_odh-ai-gateway-infra(bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf" podUID="bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78"
Oct 30 20:00:17.164987 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:17.164965    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px" event={"ID":"224aad14-bda8-4fbc-a79b-ea90309b5227","Type":"ContainerStarted","Data":"48bc7ba163aae215101abc702ee64f926e1ebdea1163b79f9862b13788a15b0b"}
Oct 30 20:00:17.166768 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:17.166748    6030 generic.go:356] "Generic (PLEG): container finished" podID="267c8f37-616e-4b1e-91ee-804399ee4b9f" containerID="a3884bb997e155804bf6bfef5ecbeb805050f3f8d35d4fdab8dd263baaf1a57a" exitCode=60
Oct 30 20:00:17.166864 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:17.166815    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww" event={"ID":"267c8f37-616e-4b1e-91ee-804399ee4b9f","Type":"ContainerDied","Data":"a3884bb997e155804bf6bfef5ecbeb805050f3f8d35d4fdab8dd263baaf1a57a"}
Oct 30 20:00:17.167160 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:17.167143    6030 scope.go:122] "RemoveContainer" containerID="a3884bb997e155804bf6bfef5ecbeb805050f3f8d35d4fdab8dd263baaf1a57a"
Oct 30 20:00:17.167459 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:17.167435    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww_odh-ai-gateway-infra(267c8f37-616e-4b1e-91ee-804399ee4b9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww" podUID="267c8f37-616e-4b1e-91ee-804399ee4b9f"
Oct 30 20:00:17.169232 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:17.169206    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs" event={"ID":"f06fcfd9-153b-402a-bce4-c64373fcdb60","Type":"ContainerStarted","Data":"c7ac0764bec55eb81b354ee29d7f2393c1a8fb5cf68856c371c02fbc5c916b77"}
Oct 30 20:00:17.177889 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:17.177867    6030 scope.go:122] "RemoveContainer" containerID="2d005189b075b35bec290d3f998ba719484ec3e83cbb883511ab1209e3b0f1b4"
Oct 30 20:00:17.192611 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:17.192588    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww"]
Oct 30 20:00:17.219974 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:17.219929    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px" podStartSLOduration=17.219916661 podStartE2EDuration="17.219916661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:17.218695619 +0000 UTC m=+1696.134302476" watchObservedRunningTime="2026-10-30 20:00:17.219916661 +0000 UTC m=+1696.135523517"
Oct 30 20:00:17.228117 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:17.228086    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf"]
Oct 30 20:00:18.180900 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.180872    6030 generic.go:356] "Generic (PLEG): container finished" podID="f06fcfd9-153b-402a-bce4-c64373fcdb60" containerID="c7ac0764bec55eb81b354ee29d7f2393c1a8fb5cf68856c371c02fbc5c916b77" exitCode=60
Oct 30 20:00:18.181326 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.180953    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs" event={"ID":"f06fcfd9-153b-402a-bce4-c64373fcdb60","Type":"ContainerDied","Data":"c7ac0764bec55eb81b354ee29d7f2393c1a8fb5cf68856c371c02fbc5c916b77"}
Oct 30 20:00:18.181326 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.181035    6030 scope.go:122] "RemoveContainer" containerID="b02f5f87e0bae542c7027581a38ac72c901bf7d498ed0c2c63087d5cb7a19a73"
Oct 30 20:00:18.181445 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.181368    6030 scope.go:122] "RemoveContainer" containerID="c7ac0764bec55eb81b354ee29d7f2393c1a8fb5cf68856c371c02fbc5c916b77"
Oct 30 20:00:18.182484 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:18.182441    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs_odh-ai-gateway-infra(f06fcfd9-153b-402a-bce4-c64373fcdb60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs" podUID="f06fcfd9-153b-402a-bce4-c64373fcdb60"
Oct 30 20:00:18.199298 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.195162    6030 scope.go:122] "RemoveContainer" containerID="7db9de846788895eb46b627f998aa6bdce49a27a31a851360b6cef10910f2da0"
Oct 30 20:00:18.199298 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.196153    6030 generic.go:356] "Generic (PLEG): container finished" podID="2cc80851-9204-48e4-a739-3d7399e6449a" containerID="43c7a0fd61aa151a23559672839d1eaff8e35b146b6755fb6a96f3c68d5efdfe" exitCode=60
Oct 30 20:00:18.199298 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.196177    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr" event={"ID":"2cc80851-9204-48e4-a739-3d7399e6449a","Type":"ContainerDied","Data":"43c7a0fd61aa151a23559672839d1eaff8e35b146b6755fb6a96f3c68d5efdfe"}
Oct 30 20:00:18.199298 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.196534    6030 scope.go:122] "RemoveContainer" containerID="43c7a0fd61aa151a23559672839d1eaff8e35b146b6755fb6a96f3c68d5efdfe"
Oct 30 20:00:18.199298 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:18.196787    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr_odh-ai-gateway-infra(2cc80851-9204-48e4-a739-3d7399e6449a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr" podUID="2cc80851-9204-48e4-a739-3d7399e6449a"
Oct 30 20:00:18.209335 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.209308    6030 generic.go:356] "Generic (PLEG): container finished" podID="224aad14-bda8-4fbc-a79b-ea90309b5227" containerID="48bc7ba163aae215101abc702ee64f926e1ebdea1163b79f9862b13788a15b0b" exitCode=60
Oct 30 20:00:18.209498 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.209364    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px" event={"ID":"224aad14-bda8-4fbc-a79b-ea90309b5227","Type":"ContainerDied","Data":"48bc7ba163aae215101abc702ee64f926e1ebdea1163b79f9862b13788a15b0b"}
Oct 30 20:00:18.209742 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.209726    6030 scope.go:122] "RemoveContainer" containerID="48bc7ba163aae215101abc702ee64f926e1ebdea1163b79f9862b13788a15b0b"
Oct 30 20:00:18.210075 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:18.210032    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px_odh-ai-gateway-infra(224aad14-bda8-4fbc-a79b-ea90309b5227)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px" podUID="224aad14-bda8-4fbc-a79b-ea90309b5227"
Oct 30 20:00:18.214278 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.214254    6030 scope.go:122] "RemoveContainer" containerID="cba4b71d7cb723df8d14b7f51d91b5f051202eef0f1a49b6ce7c3b807e54aca7"
Oct 30 20:00:18.216623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.216062    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr"]
Oct 30 20:00:18.239091 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.238732    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px"]
Oct 30 20:00:18.260317 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.260201    6030 scope.go:122] "RemoveContainer" containerID="4d15f91016098fe2aa32382c1c5645f3798b32f72628bf58b80dc2dc941424dd"
Oct 30 20:00:18.294691 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.292263    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs"]
Oct 30 20:00:18.395394 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.395368    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww"
Oct 30 20:00:18.450807 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.450740    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf"
Oct 30 20:00:18.485911 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.485880    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267c8f37-616e-4b1e-91ee-804399ee4b9f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/267c8f37-616e-4b1e-91ee-804399ee4b9f-maas-api-service-ca\") pod \"267c8f37-616e-4b1e-91ee-804399ee4b9f\" (UID: \"267c8f37-616e-4b1e-91ee-804399ee4b9f\") "
Oct 30 20:00:18.486028 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.485978    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267c8f37-616e-4b1e-91ee-804399ee4b9f-kube-api-access-n9tsj\" (UniqueName: \"kubernetes.io/projected/267c8f37-616e-4b1e-91ee-804399ee4b9f-kube-api-access-n9tsj\") pod \"267c8f37-616e-4b1e-91ee-804399ee4b9f\" (UID: \"267c8f37-616e-4b1e-91ee-804399ee4b9f\") "
Oct 30 20:00:18.486237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.486204    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267c8f37-616e-4b1e-91ee-804399ee4b9f-maas-api-service-ca" pod "267c8f37-616e-4b1e-91ee-804399ee4b9f" (UID: "267c8f37-616e-4b1e-91ee-804399ee4b9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:18.486835 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.486815    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/267c8f37-616e-4b1e-91ee-804399ee4b9f-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.488336 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.488314    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267c8f37-616e-4b1e-91ee-804399ee4b9f-kube-api-access-n9tsj" pod "267c8f37-616e-4b1e-91ee-804399ee4b9f" (UID: "267c8f37-616e-4b1e-91ee-804399ee4b9f"). InnerVolumeSpecName "kube-api-access-n9tsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:18.587940 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.587903    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78-kube-api-access-hl66f\" (UniqueName: \"kubernetes.io/projected/bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78-kube-api-access-hl66f\") pod \"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78\" (UID: \"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78\") "
Oct 30 20:00:18.588157 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.587977    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78-maas-api-service-ca\") pod \"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78\" (UID: \"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78\") "
Oct 30 20:00:18.588370 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.588348    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78-maas-api-service-ca" pod "bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78" (UID: "bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:18.588614 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.588601    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9tsj\" (UniqueName: \"kubernetes.io/projected/267c8f37-616e-4b1e-91ee-804399ee4b9f-kube-api-access-n9tsj\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.588664 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.588616    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.590303 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.590277    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78-kube-api-access-hl66f" pod "bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78" (UID: "bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78"). InnerVolumeSpecName "kube-api-access-hl66f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:18.689947 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:18.689844    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl66f\" (UniqueName: \"kubernetes.io/projected/bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78-kube-api-access-hl66f\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.219262 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.219234    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww"
Oct 30 20:00:19.219668 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.219255    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww" event={"ID":"267c8f37-616e-4b1e-91ee-804399ee4b9f","Type":"ContainerDied","Data":"41ddb7e1e83fc9a807cea6e5695195c9ee36f0c0e01b5ba8baaa3d150897a8ab"}
Oct 30 20:00:19.219668 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.219312    6030 scope.go:122] "RemoveContainer" containerID="a3884bb997e155804bf6bfef5ecbeb805050f3f8d35d4fdab8dd263baaf1a57a"
Oct 30 20:00:19.221971 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.221948    6030 generic.go:356] "Generic (PLEG): container finished" podID="e58ffa2d-b351-4239-9921-26d669955e92" containerID="b860120260b4e9942e9d7669c8e511341b2fcba4694c15c5ef2b6d4f87a8ba1a" exitCode=60
Oct 30 20:00:19.222904 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.222027    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h" event={"ID":"e58ffa2d-b351-4239-9921-26d669955e92","Type":"ContainerDied","Data":"b860120260b4e9942e9d7669c8e511341b2fcba4694c15c5ef2b6d4f87a8ba1a"}
Oct 30 20:00:19.222904 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.222426    6030 scope.go:122] "RemoveContainer" containerID="b860120260b4e9942e9d7669c8e511341b2fcba4694c15c5ef2b6d4f87a8ba1a"
Oct 30 20:00:19.222904 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:19.222706    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h_odh-ai-gateway-infra(e58ffa2d-b351-4239-9921-26d669955e92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h" podUID="e58ffa2d-b351-4239-9921-26d669955e92"
Oct 30 20:00:19.233812 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.233793    6030 scope.go:122] "RemoveContainer" containerID="7db9de846788895eb46b627f998aa6bdce49a27a31a851360b6cef10910f2da0"
Oct 30 20:00:19.241694 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.238755    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf"
Oct 30 20:00:19.241694 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.238774    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf" event={"ID":"bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78","Type":"ContainerDied","Data":"04c336e66bcd559c1d6782df22950452fcc4c96cb5138c578202e0283292b23e"}
Oct 30 20:00:19.254992 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.254970    6030 scope.go:122] "RemoveContainer" containerID="5a4e143a37f24d2c1c263e186ea341d3bb42999bb1194684cad722c8b2cc1547"
Oct 30 20:00:19.285718 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.285685    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww"]
Oct 30 20:00:19.290348 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.290314    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889840-lntww"]
Oct 30 20:00:19.387540 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.387512    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf"]
Oct 30 20:00:19.405683 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.405633    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889840-j74cf"]
Oct 30 20:00:19.469376 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.469345    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px"
Oct 30 20:00:19.472917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.472893    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs"
Oct 30 20:00:19.494771 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.494747    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr"
Oct 30 20:00:19.523716 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.521058    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/224aad14-bda8-4fbc-a79b-ea90309b5227-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/224aad14-bda8-4fbc-a79b-ea90309b5227-maas-api-service-ca\") pod \"224aad14-bda8-4fbc-a79b-ea90309b5227\" (UID: \"224aad14-bda8-4fbc-a79b-ea90309b5227\") "
Oct 30 20:00:19.523716 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.521107    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06fcfd9-153b-402a-bce4-c64373fcdb60-kube-api-access-hw62x\" (UniqueName: \"kubernetes.io/projected/f06fcfd9-153b-402a-bce4-c64373fcdb60-kube-api-access-hw62x\") pod \"f06fcfd9-153b-402a-bce4-c64373fcdb60\" (UID: \"f06fcfd9-153b-402a-bce4-c64373fcdb60\") "
Oct 30 20:00:19.523716 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.521163    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06fcfd9-153b-402a-bce4-c64373fcdb60-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06fcfd9-153b-402a-bce4-c64373fcdb60-maas-api-service-ca\") pod \"f06fcfd9-153b-402a-bce4-c64373fcdb60\" (UID: \"f06fcfd9-153b-402a-bce4-c64373fcdb60\") "
Oct 30 20:00:19.523716 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.521251    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cc80851-9204-48e4-a739-3d7399e6449a-kube-api-access-cvwt4\" (UniqueName: \"kubernetes.io/projected/2cc80851-9204-48e4-a739-3d7399e6449a-kube-api-access-cvwt4\") pod \"2cc80851-9204-48e4-a739-3d7399e6449a\" (UID: \"2cc80851-9204-48e4-a739-3d7399e6449a\") "
Oct 30 20:00:19.523716 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.521322    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cc80851-9204-48e4-a739-3d7399e6449a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc80851-9204-48e4-a739-3d7399e6449a-maas-api-service-ca\") pod \"2cc80851-9204-48e4-a739-3d7399e6449a\" (UID: \"2cc80851-9204-48e4-a739-3d7399e6449a\") "
Oct 30 20:00:19.523716 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.521390    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/224aad14-bda8-4fbc-a79b-ea90309b5227-kube-api-access-2zb6z\" (UniqueName: \"kubernetes.io/projected/224aad14-bda8-4fbc-a79b-ea90309b5227-kube-api-access-2zb6z\") pod \"224aad14-bda8-4fbc-a79b-ea90309b5227\" (UID: \"224aad14-bda8-4fbc-a79b-ea90309b5227\") "
Oct 30 20:00:19.523716 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.521563    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/224aad14-bda8-4fbc-a79b-ea90309b5227-maas-api-service-ca" pod "224aad14-bda8-4fbc-a79b-ea90309b5227" (UID: "224aad14-bda8-4fbc-a79b-ea90309b5227"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:19.523716 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.522252    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06fcfd9-153b-402a-bce4-c64373fcdb60-maas-api-service-ca" pod "f06fcfd9-153b-402a-bce4-c64373fcdb60" (UID: "f06fcfd9-153b-402a-bce4-c64373fcdb60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:19.525622 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.525585    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc80851-9204-48e4-a739-3d7399e6449a-maas-api-service-ca" pod "2cc80851-9204-48e4-a739-3d7399e6449a" (UID: "2cc80851-9204-48e4-a739-3d7399e6449a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:19.525924 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.525894    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/224aad14-bda8-4fbc-a79b-ea90309b5227-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.526130 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.525927    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06fcfd9-153b-402a-bce4-c64373fcdb60-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.526130 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.525942    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc80851-9204-48e4-a739-3d7399e6449a-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.526913 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.526887    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc80851-9204-48e4-a739-3d7399e6449a-kube-api-access-cvwt4" pod "2cc80851-9204-48e4-a739-3d7399e6449a" (UID: "2cc80851-9204-48e4-a739-3d7399e6449a"). InnerVolumeSpecName "kube-api-access-cvwt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:19.528396 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.528359    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224aad14-bda8-4fbc-a79b-ea90309b5227-kube-api-access-2zb6z" pod "224aad14-bda8-4fbc-a79b-ea90309b5227" (UID: "224aad14-bda8-4fbc-a79b-ea90309b5227"). InnerVolumeSpecName "kube-api-access-2zb6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:19.530253 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.530229    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06fcfd9-153b-402a-bce4-c64373fcdb60-kube-api-access-hw62x" pod "f06fcfd9-153b-402a-bce4-c64373fcdb60" (UID: "f06fcfd9-153b-402a-bce4-c64373fcdb60"). InnerVolumeSpecName "kube-api-access-hw62x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:19.627610 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.627576    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zb6z\" (UniqueName: \"kubernetes.io/projected/224aad14-bda8-4fbc-a79b-ea90309b5227-kube-api-access-2zb6z\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.627610 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.627605    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw62x\" (UniqueName: \"kubernetes.io/projected/f06fcfd9-153b-402a-bce4-c64373fcdb60-kube-api-access-hw62x\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.627610 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:19.627614    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvwt4\" (UniqueName: \"kubernetes.io/projected/2cc80851-9204-48e4-a739-3d7399e6449a-kube-api-access-cvwt4\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.246853 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.246824    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px"
Oct 30 20:00:20.246853 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.246842    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px" event={"ID":"224aad14-bda8-4fbc-a79b-ea90309b5227","Type":"ContainerDied","Data":"8e9fe096356687724d524e70394eab93a58e50b4d5c9de46b8631731a5417164"}
Oct 30 20:00:20.247397 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.246904    6030 scope.go:122] "RemoveContainer" containerID="48bc7ba163aae215101abc702ee64f926e1ebdea1163b79f9862b13788a15b0b"
Oct 30 20:00:20.252101 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.252080    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs"
Oct 30 20:00:20.252101 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.252079    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs" event={"ID":"f06fcfd9-153b-402a-bce4-c64373fcdb60","Type":"ContainerDied","Data":"096fb54078ac5ddbfb5e35a81c6207ce351d17d1d20216dbaf94716896e87f7a"}
Oct 30 20:00:20.253712 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.253688    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr"
Oct 30 20:00:20.253712 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.253701    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr" event={"ID":"2cc80851-9204-48e4-a739-3d7399e6449a","Type":"ContainerDied","Data":"b537d0904ee88907381242f83eb4c14672bed1e97564078e09f8f08ebba8cb79"}
Oct 30 20:00:20.259583 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.259567    6030 scope.go:122] "RemoveContainer" containerID="c7ac0764bec55eb81b354ee29d7f2393c1a8fb5cf68856c371c02fbc5c916b77"
Oct 30 20:00:20.269015 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.268997    6030 scope.go:122] "RemoveContainer" containerID="43c7a0fd61aa151a23559672839d1eaff8e35b146b6755fb6a96f3c68d5efdfe"
Oct 30 20:00:20.284505 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.284481    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px"]
Oct 30 20:00:20.287438 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.287418    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889840-244px"]
Oct 30 20:00:20.302344 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.302322    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr"]
Oct 30 20:00:20.313463 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.313440    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889840-8hvbr"]
Oct 30 20:00:20.330577 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.330552    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h"]
Oct 30 20:00:20.333332 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.333311    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs"]
Oct 30 20:00:20.341346 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.341320    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889840-9qdjs"]
Oct 30 20:00:20.499685 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.499626    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h"
Oct 30 20:00:20.553435 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.544535    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e58ffa2d-b351-4239-9921-26d669955e92-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58ffa2d-b351-4239-9921-26d669955e92-maas-api-service-ca\") pod \"e58ffa2d-b351-4239-9921-26d669955e92\" (UID: \"e58ffa2d-b351-4239-9921-26d669955e92\") "
Oct 30 20:00:20.553435 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.545118    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58ffa2d-b351-4239-9921-26d669955e92-kube-api-access-ct2s7\" (UniqueName: \"kubernetes.io/projected/e58ffa2d-b351-4239-9921-26d669955e92-kube-api-access-ct2s7\") pod \"e58ffa2d-b351-4239-9921-26d669955e92\" (UID: \"e58ffa2d-b351-4239-9921-26d669955e92\") "
Oct 30 20:00:20.553435 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.544902    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e58ffa2d-b351-4239-9921-26d669955e92-maas-api-service-ca" pod "e58ffa2d-b351-4239-9921-26d669955e92" (UID: "e58ffa2d-b351-4239-9921-26d669955e92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:20.578334 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.572225    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58ffa2d-b351-4239-9921-26d669955e92-kube-api-access-ct2s7" pod "e58ffa2d-b351-4239-9921-26d669955e92" (UID: "e58ffa2d-b351-4239-9921-26d669955e92"). InnerVolumeSpecName "kube-api-access-ct2s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:20.655748 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.655691    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct2s7\" (UniqueName: \"kubernetes.io/projected/e58ffa2d-b351-4239-9921-26d669955e92-kube-api-access-ct2s7\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.656035 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:20.656012    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58ffa2d-b351-4239-9921-26d669955e92-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:21.206028 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:21.205990    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="224aad14-bda8-4fbc-a79b-ea90309b5227" path="/var/lib/kubelet/pods/224aad14-bda8-4fbc-a79b-ea90309b5227/volumes"
Oct 30 20:00:21.207602 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:21.207565    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267c8f37-616e-4b1e-91ee-804399ee4b9f" path="/var/lib/kubelet/pods/267c8f37-616e-4b1e-91ee-804399ee4b9f/volumes"
Oct 30 20:00:21.208179 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:21.208161    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cc80851-9204-48e4-a739-3d7399e6449a" path="/var/lib/kubelet/pods/2cc80851-9204-48e4-a739-3d7399e6449a/volumes"
Oct 30 20:00:21.209031 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:21.209009    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78" path="/var/lib/kubelet/pods/bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78/volumes"
Oct 30 20:00:21.209647 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:21.209624    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06fcfd9-153b-402a-bce4-c64373fcdb60" path="/var/lib/kubelet/pods/f06fcfd9-153b-402a-bce4-c64373fcdb60/volumes"
Oct 30 20:00:21.264266 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:21.264229    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h"
Oct 30 20:00:21.264731 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:21.264230    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h" event={"ID":"e58ffa2d-b351-4239-9921-26d669955e92","Type":"ContainerDied","Data":"e424d00cdbede5e518dc46db312804bf363f800993f695a8a68e08503e2f04c0"}
Oct 30 20:00:21.264731 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:21.264373    6030 scope.go:122] "RemoveContainer" containerID="b860120260b4e9942e9d7669c8e511341b2fcba4694c15c5ef2b6d4f87a8ba1a"
Oct 30 20:00:21.290360 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:21.290336    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h"]
Oct 30 20:00:21.294574 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:21.294549    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889840-zmw9h"]
Oct 30 20:00:23.203129 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:00:23.203102    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e58ffa2d-b351-4239-9921-26d669955e92" path="/var/lib/kubelet/pods/e58ffa2d-b351-4239-9921-26d669955e92/volumes"
Oct 30 20:00:34.288651 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:00:34.288602    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:01:01.114066 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:01.113904    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"]
Oct 30 20:01:04.290375 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:01:04.290334    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:01:16.147319 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.147267    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w" podUID="22dadc59-4ea6-4f6d-84e7-32735402c133" containerName="main" containerID="cri-o://0e49588925c9fd60390197df12dc10f027d969ee8250efa7ad69bf950b85c0f4" gracePeriod=45
Oct 30 20:01:16.514176 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.514152    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 20:01:16.594804 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.594764    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dadc59-4ea6-4f6d-84e7-32735402c133-kube-api-access-qr9pn\" (UniqueName: \"kubernetes.io/projected/22dadc59-4ea6-4f6d-84e7-32735402c133-kube-api-access-qr9pn\") pod \"22dadc59-4ea6-4f6d-84e7-32735402c133\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") "
Oct 30 20:01:16.594971 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.594815    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-dshm\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-dshm\") pod \"22dadc59-4ea6-4f6d-84e7-32735402c133\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") "
Oct 30 20:01:16.594971 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.594901    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-home\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-home\") pod \"22dadc59-4ea6-4f6d-84e7-32735402c133\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") "
Oct 30 20:01:16.595100 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.595083    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-tmp-dir\") pod \"22dadc59-4ea6-4f6d-84e7-32735402c133\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") "
Oct 30 20:01:16.595144 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.595131    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22dadc59-4ea6-4f6d-84e7-32735402c133-tls-certs\" (UniqueName: \"kubernetes.io/secret/22dadc59-4ea6-4f6d-84e7-32735402c133-tls-certs\") pod \"22dadc59-4ea6-4f6d-84e7-32735402c133\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") "
Oct 30 20:01:16.595209 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.595181    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-model-cache\") pod \"22dadc59-4ea6-4f6d-84e7-32735402c133\" (UID: \"22dadc59-4ea6-4f6d-84e7-32735402c133\") "
Oct 30 20:01:16.595269 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.595237    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-home" pod "22dadc59-4ea6-4f6d-84e7-32735402c133" (UID: "22dadc59-4ea6-4f6d-84e7-32735402c133"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:16.595323 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.595281    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-tmp-dir" pod "22dadc59-4ea6-4f6d-84e7-32735402c133" (UID: "22dadc59-4ea6-4f6d-84e7-32735402c133"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:16.595669 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.595467    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-model-cache" pod "22dadc59-4ea6-4f6d-84e7-32735402c133" (UID: "22dadc59-4ea6-4f6d-84e7-32735402c133"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:16.595783 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.595767    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:16.595848 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.595785    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:16.595848 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.595793    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:16.597250 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.597229    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dadc59-4ea6-4f6d-84e7-32735402c133-kube-api-access-qr9pn" pod "22dadc59-4ea6-4f6d-84e7-32735402c133" (UID: "22dadc59-4ea6-4f6d-84e7-32735402c133"). InnerVolumeSpecName "kube-api-access-qr9pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:16.597250 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.597243    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-dshm" pod "22dadc59-4ea6-4f6d-84e7-32735402c133" (UID: "22dadc59-4ea6-4f6d-84e7-32735402c133"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:16.597391 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.597367    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22dadc59-4ea6-4f6d-84e7-32735402c133-tls-certs" pod "22dadc59-4ea6-4f6d-84e7-32735402c133" (UID: "22dadc59-4ea6-4f6d-84e7-32735402c133"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:16.697751 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.697656    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr9pn\" (UniqueName: \"kubernetes.io/projected/22dadc59-4ea6-4f6d-84e7-32735402c133-kube-api-access-qr9pn\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:16.697751 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.697688    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22dadc59-4ea6-4f6d-84e7-32735402c133-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:16.697751 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.697702    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22dadc59-4ea6-4f6d-84e7-32735402c133-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:16.911070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.910761    6030 generic.go:356] "Generic (PLEG): container finished" podID="22dadc59-4ea6-4f6d-84e7-32735402c133" containerID="0e49588925c9fd60390197df12dc10f027d969ee8250efa7ad69bf950b85c0f4" exitCode=0
Oct 30 20:01:16.911070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.910976    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w" event={"ID":"22dadc59-4ea6-4f6d-84e7-32735402c133","Type":"ContainerDied","Data":"0e49588925c9fd60390197df12dc10f027d969ee8250efa7ad69bf950b85c0f4"}
Oct 30 20:01:16.911070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.911017    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w" event={"ID":"22dadc59-4ea6-4f6d-84e7-32735402c133","Type":"ContainerDied","Data":"9113a841668556f7fa66d9f6a48680a17417104d847f23437e21810d1225152a"}
Oct 30 20:01:16.911070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.911059    6030 scope.go:122] "RemoveContainer" containerID="0e49588925c9fd60390197df12dc10f027d969ee8250efa7ad69bf950b85c0f4"
Oct 30 20:01:16.911426 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:16.911282    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"
Oct 30 20:01:17.026952 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:17.014228    6030 scope.go:122] "RemoveContainer" containerID="0e49588925c9fd60390197df12dc10f027d969ee8250efa7ad69bf950b85c0f4"
Oct 30 20:01:17.026952 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:17.024217    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"]
Oct 30 20:01:17.038692 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:01:17.028651    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e49588925c9fd60390197df12dc10f027d969ee8250efa7ad69bf950b85c0f4\": container with ID starting with 0e49588925c9fd60390197df12dc10f027d969ee8250efa7ad69bf950b85c0f4 not found: ID does not exist" containerID="0e49588925c9fd60390197df12dc10f027d969ee8250efa7ad69bf950b85c0f4"
Oct 30 20:01:17.038692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:17.028701    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e49588925c9fd60390197df12dc10f027d969ee8250efa7ad69bf950b85c0f4"} err="failed to get container status \"0e49588925c9fd60390197df12dc10f027d969ee8250efa7ad69bf950b85c0f4\": rpc error: code = NotFound desc = could not find container \"0e49588925c9fd60390197df12dc10f027d969ee8250efa7ad69bf950b85c0f4\": container with ID starting with 0e49588925c9fd60390197df12dc10f027d969ee8250efa7ad69bf950b85c0f4 not found: ID does not exist"
Oct 30 20:01:17.038692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:17.030601    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0eeba5/auth-test-model-0eeba5-kserve-5f4d56485-r9p9w"]
Oct 30 20:01:17.203508 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:17.203473    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22dadc59-4ea6-4f6d-84e7-32735402c133" path="/var/lib/kubelet/pods/22dadc59-4ea6-4f6d-84e7-32735402c133/volumes"
Oct 30 20:01:25.142899 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.142855    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"]
Oct 30 20:01:25.144235 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144209    6030 cpu_manager.go:414] "containerMap: removing container" podUID="f06fcfd9-153b-402a-bce4-c64373fcdb60" containerName="cleanup"
Oct 30 20:01:25.144420 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144404    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06fcfd9-153b-402a-bce4-c64373fcdb60" containerName="cleanup"
Oct 30 20:01:25.144537 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144526    6030 cpu_manager.go:414] "containerMap: removing container" podUID="2cc80851-9204-48e4-a739-3d7399e6449a" containerName="cleanup"
Oct 30 20:01:25.144610 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144600    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc80851-9204-48e4-a739-3d7399e6449a" containerName="cleanup"
Oct 30 20:01:25.144695 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144676    6030 cpu_manager.go:414] "containerMap: removing container" podUID="22dadc59-4ea6-4f6d-84e7-32735402c133" containerName="main"
Oct 30 20:01:25.144695 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144693    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dadc59-4ea6-4f6d-84e7-32735402c133" containerName="main"
Oct 30 20:01:25.144877 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144719    6030 cpu_manager.go:414] "containerMap: removing container" podUID="bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78" containerName="cleanup"
Oct 30 20:01:25.144877 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144727    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78" containerName="cleanup"
Oct 30 20:01:25.144877 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144743    6030 cpu_manager.go:414] "containerMap: removing container" podUID="224aad14-bda8-4fbc-a79b-ea90309b5227" containerName="cleanup"
Oct 30 20:01:25.144877 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144749    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="224aad14-bda8-4fbc-a79b-ea90309b5227" containerName="cleanup"
Oct 30 20:01:25.144877 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144761    6030 cpu_manager.go:414] "containerMap: removing container" podUID="f06fcfd9-153b-402a-bce4-c64373fcdb60" containerName="cleanup"
Oct 30 20:01:25.144877 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144776    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06fcfd9-153b-402a-bce4-c64373fcdb60" containerName="cleanup"
Oct 30 20:01:25.144877 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144801    6030 cpu_manager.go:414] "containerMap: removing container" podUID="bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78" containerName="cleanup"
Oct 30 20:01:25.144877 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144808    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78" containerName="cleanup"
Oct 30 20:01:25.144877 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144822    6030 cpu_manager.go:414] "containerMap: removing container" podUID="e58ffa2d-b351-4239-9921-26d669955e92" containerName="cleanup"
Oct 30 20:01:25.144877 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144834    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58ffa2d-b351-4239-9921-26d669955e92" containerName="cleanup"
Oct 30 20:01:25.144877 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144843    6030 cpu_manager.go:414] "containerMap: removing container" podUID="224aad14-bda8-4fbc-a79b-ea90309b5227" containerName="cleanup"
Oct 30 20:01:25.144877 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144850    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="224aad14-bda8-4fbc-a79b-ea90309b5227" containerName="cleanup"
Oct 30 20:01:25.144877 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144864    6030 cpu_manager.go:414] "containerMap: removing container" podUID="2cc80851-9204-48e4-a739-3d7399e6449a" containerName="cleanup"
Oct 30 20:01:25.144877 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144875    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc80851-9204-48e4-a739-3d7399e6449a" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144887    6030 cpu_manager.go:414] "containerMap: removing container" podUID="bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144894    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144911    6030 cpu_manager.go:414] "containerMap: removing container" podUID="e58ffa2d-b351-4239-9921-26d669955e92" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144918    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58ffa2d-b351-4239-9921-26d669955e92" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144937    6030 cpu_manager.go:414] "containerMap: removing container" podUID="267c8f37-616e-4b1e-91ee-804399ee4b9f" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144944    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267c8f37-616e-4b1e-91ee-804399ee4b9f" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144976    6030 cpu_manager.go:414] "containerMap: removing container" podUID="e58ffa2d-b351-4239-9921-26d669955e92" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.144983    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58ffa2d-b351-4239-9921-26d669955e92" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145058    6030 cpu_manager.go:414] "containerMap: removing container" podUID="224aad14-bda8-4fbc-a79b-ea90309b5227" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145065    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="224aad14-bda8-4fbc-a79b-ea90309b5227" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145083    6030 cpu_manager.go:414] "containerMap: removing container" podUID="267c8f37-616e-4b1e-91ee-804399ee4b9f" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145090    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267c8f37-616e-4b1e-91ee-804399ee4b9f" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145112    6030 cpu_manager.go:414] "containerMap: removing container" podUID="267c8f37-616e-4b1e-91ee-804399ee4b9f" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145119    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267c8f37-616e-4b1e-91ee-804399ee4b9f" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145396    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06fcfd9-153b-402a-bce4-c64373fcdb60" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145425    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="224aad14-bda8-4fbc-a79b-ea90309b5227" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145435    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="267c8f37-616e-4b1e-91ee-804399ee4b9f" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145452    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc80851-9204-48e4-a739-3d7399e6449a" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145463    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="267c8f37-616e-4b1e-91ee-804399ee4b9f" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145480    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06fcfd9-153b-402a-bce4-c64373fcdb60" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145490    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dadc59-4ea6-4f6d-84e7-32735402c133" containerName="main"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145506    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145521    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="224aad14-bda8-4fbc-a79b-ea90309b5227" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145537    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06fcfd9-153b-402a-bce4-c64373fcdb60" containerName="cleanup"
Oct 30 20:01:25.145542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145552    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78" containerName="cleanup"
Oct 30 20:01:25.146677 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145561    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58ffa2d-b351-4239-9921-26d669955e92" containerName="cleanup"
Oct 30 20:01:25.146677 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145577    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="224aad14-bda8-4fbc-a79b-ea90309b5227" containerName="cleanup"
Oct 30 20:01:25.146677 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145585    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc80851-9204-48e4-a739-3d7399e6449a" containerName="cleanup"
Oct 30 20:01:25.146677 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145592    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc99f4e-ef07-48b6-a8fd-f01fc99b3f78" containerName="cleanup"
Oct 30 20:01:25.146677 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.145605    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58ffa2d-b351-4239-9921-26d669955e92" containerName="cleanup"
Oct 30 20:01:25.151908 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.151886    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"
Oct 30 20:01:25.163433 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.163407    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c9bce8\""
Oct 30 20:01:25.165710 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.165689    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"]
Oct 30 20:01:25.169177 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.169154    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"]
Oct 30 20:01:25.169776 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.169756    6030 cpu_manager.go:414] "containerMap: removing container" podUID="f06fcfd9-153b-402a-bce4-c64373fcdb60" containerName="cleanup"
Oct 30 20:01:25.169857 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.169778    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06fcfd9-153b-402a-bce4-c64373fcdb60" containerName="cleanup"
Oct 30 20:01:25.169857 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.169802    6030 cpu_manager.go:414] "containerMap: removing container" podUID="2cc80851-9204-48e4-a739-3d7399e6449a" containerName="cleanup"
Oct 30 20:01:25.169857 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.169810    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc80851-9204-48e4-a739-3d7399e6449a" containerName="cleanup"
Oct 30 20:01:25.170290 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.169957    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc80851-9204-48e4-a739-3d7399e6449a" containerName="cleanup"
Oct 30 20:01:25.170290 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.169969    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58ffa2d-b351-4239-9921-26d669955e92" containerName="cleanup"
Oct 30 20:01:25.170290 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.169992    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="267c8f37-616e-4b1e-91ee-804399ee4b9f" containerName="cleanup"
Oct 30 20:01:25.173631 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.173609    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"
Oct 30 20:01:25.208089 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.206719    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"]
Oct 30 20:01:25.255458 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.255428    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c9bce8-dockercfg-qnj7l\""
Oct 30 20:01:25.372092 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.369303    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvxsw\" (UniqueName: \"kubernetes.io/projected/aff9109e-c1a8-4ba0-9510-dc8d18be9547-kube-api-access-jvxsw\") pod \"payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg\" (UID: \"aff9109e-c1a8-4ba0-9510-dc8d18be9547\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"
Oct 30 20:01:25.372092 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.369375    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4kd7\" (UniqueName: \"kubernetes.io/projected/2c8b6e10-33d3-479c-b3ae-c43d2058022e-kube-api-access-p4kd7\") pod \"payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b\" (UID: \"2c8b6e10-33d3-479c-b3ae-c43d2058022e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"
Oct 30 20:01:25.372092 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.369422    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff9109e-c1a8-4ba0-9510-dc8d18be9547-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg\" (UID: \"aff9109e-c1a8-4ba0-9510-dc8d18be9547\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"
Oct 30 20:01:25.372092 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.369493    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8b6e10-33d3-479c-b3ae-c43d2058022e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b\" (UID: \"2c8b6e10-33d3-479c-b3ae-c43d2058022e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"
Oct 30 20:01:25.471201 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.471098    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvxsw\" (UniqueName: \"kubernetes.io/projected/aff9109e-c1a8-4ba0-9510-dc8d18be9547-kube-api-access-jvxsw\") pod \"payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg\" (UID: \"aff9109e-c1a8-4ba0-9510-dc8d18be9547\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"
Oct 30 20:01:25.471201 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.471173    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4kd7\" (UniqueName: \"kubernetes.io/projected/2c8b6e10-33d3-479c-b3ae-c43d2058022e-kube-api-access-p4kd7\") pod \"payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b\" (UID: \"2c8b6e10-33d3-479c-b3ae-c43d2058022e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"
Oct 30 20:01:25.471448 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.471222    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff9109e-c1a8-4ba0-9510-dc8d18be9547-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg\" (UID: \"aff9109e-c1a8-4ba0-9510-dc8d18be9547\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"
Oct 30 20:01:25.471448 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.471268    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8b6e10-33d3-479c-b3ae-c43d2058022e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b\" (UID: \"2c8b6e10-33d3-479c-b3ae-c43d2058022e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"
Oct 30 20:01:25.472204 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.472183    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff9109e-c1a8-4ba0-9510-dc8d18be9547-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg\" (UID: \"aff9109e-c1a8-4ba0-9510-dc8d18be9547\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"
Oct 30 20:01:25.472204 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.472196    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8b6e10-33d3-479c-b3ae-c43d2058022e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b\" (UID: \"2c8b6e10-33d3-479c-b3ae-c43d2058022e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"
Oct 30 20:01:25.494810 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.494770    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvxsw\" (UniqueName: \"kubernetes.io/projected/aff9109e-c1a8-4ba0-9510-dc8d18be9547-kube-api-access-jvxsw\") pod \"payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg\" (UID: \"aff9109e-c1a8-4ba0-9510-dc8d18be9547\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"
Oct 30 20:01:25.495768 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.495741    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4kd7\" (UniqueName: \"kubernetes.io/projected/2c8b6e10-33d3-479c-b3ae-c43d2058022e-kube-api-access-p4kd7\") pod \"payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b\" (UID: \"2c8b6e10-33d3-479c-b3ae-c43d2058022e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"
Oct 30 20:01:25.768807 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.768697    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"
Oct 30 20:01:25.790747 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:25.790719    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"
Oct 30 20:01:26.095116 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:01:26.095089    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8b6e10_33d3_479c_b3ae_c43d2058022e.slice/crio-c295563fd5ff7a0124821e950c68b823950c4191f80dd8811cd32aaedf429ec2 WatchSource:0}: Error finding container c295563fd5ff7a0124821e950c68b823950c4191f80dd8811cd32aaedf429ec2: Status 404 returned error can't find the container with id c295563fd5ff7a0124821e950c68b823950c4191f80dd8811cd32aaedf429ec2
Oct 30 20:01:26.095792 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:01:26.095770    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff9109e_c1a8_4ba0_9510_dc8d18be9547.slice/crio-7297574a63896c26b000caddce33100770b9f4ede59577bc1be38407fae6f763 WatchSource:0}: Error finding container 7297574a63896c26b000caddce33100770b9f4ede59577bc1be38407fae6f763: Status 404 returned error can't find the container with id 7297574a63896c26b000caddce33100770b9f4ede59577bc1be38407fae6f763
Oct 30 20:01:26.097118 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.097096    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"]
Oct 30 20:01:26.101450 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.101426    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"]
Oct 30 20:01:26.333711 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.333682    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"]
Oct 30 20:01:26.337877 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.337857    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"
Oct 30 20:01:26.343101 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.343021    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-75b361\""
Oct 30 20:01:26.343190 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.343159    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-75b361-dockercfg-qq5xw\""
Oct 30 20:01:26.352473 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.352452    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"]
Oct 30 20:01:26.389582 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.389074    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfqpb\" (UniqueName: \"kubernetes.io/projected/ba53b339-9f04-4b57-b1f0-4ee28ba07af7-kube-api-access-vfqpb\") pod \"payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc\" (UID: \"ba53b339-9f04-4b57-b1f0-4ee28ba07af7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"
Oct 30 20:01:26.389582 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.389380    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba53b339-9f04-4b57-b1f0-4ee28ba07af7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc\" (UID: \"ba53b339-9f04-4b57-b1f0-4ee28ba07af7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"
Oct 30 20:01:26.516068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.515100    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba53b339-9f04-4b57-b1f0-4ee28ba07af7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc\" (UID: \"ba53b339-9f04-4b57-b1f0-4ee28ba07af7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"
Oct 30 20:01:26.516068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.515280    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfqpb\" (UniqueName: \"kubernetes.io/projected/ba53b339-9f04-4b57-b1f0-4ee28ba07af7-kube-api-access-vfqpb\") pod \"payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc\" (UID: \"ba53b339-9f04-4b57-b1f0-4ee28ba07af7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"
Oct 30 20:01:26.521116 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.516588    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba53b339-9f04-4b57-b1f0-4ee28ba07af7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc\" (UID: \"ba53b339-9f04-4b57-b1f0-4ee28ba07af7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"
Oct 30 20:01:26.537068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.534572    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfqpb\" (UniqueName: \"kubernetes.io/projected/ba53b339-9f04-4b57-b1f0-4ee28ba07af7-kube-api-access-vfqpb\") pod \"payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc\" (UID: \"ba53b339-9f04-4b57-b1f0-4ee28ba07af7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"
Oct 30 20:01:26.653108 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.652903    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"
Oct 30 20:01:26.944915 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.944889    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"]
Oct 30 20:01:26.946004 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:01:26.945974    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba53b339_9f04_4b57_b1f0_4ee28ba07af7.slice/crio-790f6a94e863f2d419be81bf3ce575547186f8c826c5ab03880c275757989252 WatchSource:0}: Error finding container 790f6a94e863f2d419be81bf3ce575547186f8c826c5ab03880c275757989252: Status 404 returned error can't find the container with id 790f6a94e863f2d419be81bf3ce575547186f8c826c5ab03880c275757989252
Oct 30 20:01:26.986582 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.986554    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc" event={"ID":"ba53b339-9f04-4b57-b1f0-4ee28ba07af7","Type":"ContainerStarted","Data":"790f6a94e863f2d419be81bf3ce575547186f8c826c5ab03880c275757989252"}
Oct 30 20:01:26.988289 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.988261    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b" event={"ID":"2c8b6e10-33d3-479c-b3ae-c43d2058022e","Type":"ContainerStarted","Data":"33369a845a4b39dea21c8a43bc226c714fefebd69f16a210ed69dff2af6379f0"}
Oct 30 20:01:26.988414 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.988320    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"
Oct 30 20:01:26.988414 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.988343    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b" event={"ID":"2c8b6e10-33d3-479c-b3ae-c43d2058022e","Type":"ContainerStarted","Data":"c295563fd5ff7a0124821e950c68b823950c4191f80dd8811cd32aaedf429ec2"}
Oct 30 20:01:26.989993 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.989962    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg" event={"ID":"aff9109e-c1a8-4ba0-9510-dc8d18be9547","Type":"ContainerStarted","Data":"a4e877eed84af948eba7b99ae2c76f8651a42ef07b86664a116c3dd80cca05fc"}
Oct 30 20:01:26.990108 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.990015    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg" event={"ID":"aff9109e-c1a8-4ba0-9510-dc8d18be9547","Type":"ContainerStarted","Data":"7297574a63896c26b000caddce33100770b9f4ede59577bc1be38407fae6f763"}
Oct 30 20:01:26.990163 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:26.990122    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"
Oct 30 20:01:27.015409 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:27.015368    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b" podStartSLOduration=2.01535355 podStartE2EDuration="2.01535355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:27.013306529 +0000 UTC m=+1765.928913386" watchObservedRunningTime="2026-10-30 20:01:27.01535355 +0000 UTC m=+1765.930960412"
Oct 30 20:01:27.043676 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:27.043630    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg" podStartSLOduration=2.04361704 podStartE2EDuration="2.04361704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:27.038908823 +0000 UTC m=+1765.954515679" watchObservedRunningTime="2026-10-30 20:01:27.04361704 +0000 UTC m=+1765.959223895"
Oct 30 20:01:28.008436 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:28.008401    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc" event={"ID":"ba53b339-9f04-4b57-b1f0-4ee28ba07af7","Type":"ContainerStarted","Data":"43d2634156394444edd7f85cebeaf4511a2bb0fc1a28dc5c7908a6e9f9ceda94"}
Oct 30 20:01:28.008870 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:28.008622    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"
Oct 30 20:01:28.049909 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:28.049863    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc" podStartSLOduration=2.049850003 podStartE2EDuration="2.049850003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:28.046254337 +0000 UTC m=+1766.961861192" watchObservedRunningTime="2026-10-30 20:01:28.049850003 +0000 UTC m=+1766.965456861"
Oct 30 20:01:34.289237 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:01:34.289195    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:01:38.011841 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:38.011811    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"
Oct 30 20:01:38.012361 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:38.011882    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"
Oct 30 20:01:39.019305 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:39.019267    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"
Oct 30 20:01:40.962870 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:40.962832    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"]
Oct 30 20:01:40.968792 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:40.968748    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:01:40.974844 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:40.974817    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-75b361-serving-cert\""
Oct 30 20:01:40.978067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:40.978028    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"]
Oct 30 20:01:41.053565 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:41.053531    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffrtg\" (UniqueName: \"kubernetes.io/projected/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-kube-api-access-ffrtg\") pod \"maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d\" (UID: \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:01:41.053722 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:41.053578    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d\" (UID: \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:01:41.053722 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:41.053670    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d\" (UID: \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:01:41.053894 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:41.053881    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d\" (UID: \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:01:41.155588 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:41.155553    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffrtg\" (UniqueName: \"kubernetes.io/projected/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-kube-api-access-ffrtg\") pod \"maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d\" (UID: \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:01:41.155751 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:41.155598    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d\" (UID: \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:01:41.155751 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:41.155639    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d\" (UID: \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:01:41.155888 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:41.155869    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d\" (UID: \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:01:41.156330 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:41.156308    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d\" (UID: \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:01:41.158273 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:41.158251    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d\" (UID: \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:01:41.158373 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:41.158292    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d\" (UID: \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:01:41.169614 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:41.169587    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffrtg\" (UniqueName: \"kubernetes.io/projected/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-kube-api-access-ffrtg\") pod \"maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d\" (UID: \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:01:41.288767 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:41.288682    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:01:41.603570 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:41.603457    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"]
Oct 30 20:01:41.604582 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:41.604552    6030 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:01:42.162992 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:42.162098    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d" event={"ID":"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3","Type":"ContainerStarted","Data":"2d124454ca590c1cc54dbbc98bea63c8def3859fdc9d675c6b2e124f4f046ddc"}
Oct 30 20:01:42.162992 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:42.162170    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d" event={"ID":"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3","Type":"ContainerStarted","Data":"9478157b757862571d9c40d39b01a2730110f5c322e0808f903796177c117717"}
Oct 30 20:01:42.162992 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:42.162298    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:01:42.204932 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:42.204308    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d" podStartSLOduration=1.81291148 podStartE2EDuration="2.204287147s" totalImagesPullingTime="391.375667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:42.198967748 +0000 UTC m=+1781.114574605" watchObservedRunningTime="2026-10-30 20:01:42.204287147 +0000 UTC m=+1781.119894005"
Oct 30 20:01:43.853123 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:43.839267    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"]
Oct 30 20:01:48.175715 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:48.175684    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:01:53.555933 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:53.555896    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"]
Oct 30 20:01:53.560131 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:53.560114    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"
Oct 30 20:01:53.572329 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:53.572302    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8c8241\""
Oct 30 20:01:53.573752 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:53.573725    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"]
Oct 30 20:01:53.663201 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:53.662769    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8c8241-dockercfg-b9d6j\""
Oct 30 20:01:53.723070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:53.710877    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ca4407-7903-475d-8c49-2dde2ba111b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp\" (UID: \"75ca4407-7903-475d-8c49-2dde2ba111b0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"
Oct 30 20:01:53.723070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:53.711086    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d98rz\" (UniqueName: \"kubernetes.io/projected/75ca4407-7903-475d-8c49-2dde2ba111b0-kube-api-access-d98rz\") pod \"payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp\" (UID: \"75ca4407-7903-475d-8c49-2dde2ba111b0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"
Oct 30 20:01:53.825119 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:53.824088    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ca4407-7903-475d-8c49-2dde2ba111b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp\" (UID: \"75ca4407-7903-475d-8c49-2dde2ba111b0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"
Oct 30 20:01:53.825119 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:53.824276    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d98rz\" (UniqueName: \"kubernetes.io/projected/75ca4407-7903-475d-8c49-2dde2ba111b0-kube-api-access-d98rz\") pod \"payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp\" (UID: \"75ca4407-7903-475d-8c49-2dde2ba111b0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"
Oct 30 20:01:53.838070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:53.826450    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ca4407-7903-475d-8c49-2dde2ba111b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp\" (UID: \"75ca4407-7903-475d-8c49-2dde2ba111b0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"
Oct 30 20:01:53.857410 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:53.855720    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d98rz\" (UniqueName: \"kubernetes.io/projected/75ca4407-7903-475d-8c49-2dde2ba111b0-kube-api-access-d98rz\") pod \"payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp\" (UID: \"75ca4407-7903-475d-8c49-2dde2ba111b0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"
Oct 30 20:01:53.876440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:53.876412    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"
Oct 30 20:01:54.205719 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:54.205687    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"]
Oct 30 20:01:54.206433 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:01:54.206408    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ca4407_7903_475d_8c49_2dde2ba111b0.slice/crio-09be82df00f63d67a28edf94253c9ee10cffaf2c6474f8d3fa1817427ef99a1b WatchSource:0}: Error finding container 09be82df00f63d67a28edf94253c9ee10cffaf2c6474f8d3fa1817427ef99a1b: Status 404 returned error can't find the container with id 09be82df00f63d67a28edf94253c9ee10cffaf2c6474f8d3fa1817427ef99a1b
Oct 30 20:01:54.285366 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:54.285336    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp" event={"ID":"75ca4407-7903-475d-8c49-2dde2ba111b0","Type":"ContainerStarted","Data":"09be82df00f63d67a28edf94253c9ee10cffaf2c6474f8d3fa1817427ef99a1b"}
Oct 30 20:01:55.293885 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:55.293847    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp" event={"ID":"75ca4407-7903-475d-8c49-2dde2ba111b0","Type":"ContainerStarted","Data":"711d4b63f87932b08d60f4e961aa68172822d98b2ae6437f31fb22d53d4f150f"}
Oct 30 20:01:55.294378 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:55.293930    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"
Oct 30 20:01:55.322794 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:55.322731    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp" podStartSLOduration=2.322715335 podStartE2EDuration="2.322715335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:55.319320773 +0000 UTC m=+1794.234927610" watchObservedRunningTime="2026-10-30 20:01:55.322715335 +0000 UTC m=+1794.238322194"
Oct 30 20:01:58.898485 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:58.898444    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725" podUID="864c1436-736d-4921-b137-ed7d41914a74" containerName="main" containerID="cri-o://37777689b708c751660c389a9fc6aa23d69de38ccf066cd09d969fecfeb697e2" gracePeriod=45
Oct 30 20:01:59.189259 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.189234    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 20:01:59.241620 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.241591    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-dshm\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-dshm\") pod \"864c1436-736d-4921-b137-ed7d41914a74\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") "
Oct 30 20:01:59.241798 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.241660    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-home\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-home\") pod \"864c1436-736d-4921-b137-ed7d41914a74\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") "
Oct 30 20:01:59.241798 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.241724    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-tmp-dir\") pod \"864c1436-736d-4921-b137-ed7d41914a74\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") "
Oct 30 20:01:59.241798 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.241797    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/864c1436-736d-4921-b137-ed7d41914a74-tls-certs\" (UniqueName: \"kubernetes.io/secret/864c1436-736d-4921-b137-ed7d41914a74-tls-certs\") pod \"864c1436-736d-4921-b137-ed7d41914a74\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") "
Oct 30 20:01:59.241968 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.241920    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-home" pod "864c1436-736d-4921-b137-ed7d41914a74" (UID: "864c1436-736d-4921-b137-ed7d41914a74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:59.242081 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.241931    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864c1436-736d-4921-b137-ed7d41914a74-kube-api-access-h6bkw\" (UniqueName: \"kubernetes.io/projected/864c1436-736d-4921-b137-ed7d41914a74-kube-api-access-h6bkw\") pod \"864c1436-736d-4921-b137-ed7d41914a74\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") "
Oct 30 20:01:59.242081 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.241994    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-tmp-dir" pod "864c1436-736d-4921-b137-ed7d41914a74" (UID: "864c1436-736d-4921-b137-ed7d41914a74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:59.242081 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.242029    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-model-cache\") pod \"864c1436-736d-4921-b137-ed7d41914a74\" (UID: \"864c1436-736d-4921-b137-ed7d41914a74\") "
Oct 30 20:01:59.242250 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.242221    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-model-cache" pod "864c1436-736d-4921-b137-ed7d41914a74" (UID: "864c1436-736d-4921-b137-ed7d41914a74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:59.242692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.242625    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:59.242692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.242648    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:59.242692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.242663    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:59.244190 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.244167    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-dshm" pod "864c1436-736d-4921-b137-ed7d41914a74" (UID: "864c1436-736d-4921-b137-ed7d41914a74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:59.244659 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.244638    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864c1436-736d-4921-b137-ed7d41914a74-tls-certs" pod "864c1436-736d-4921-b137-ed7d41914a74" (UID: "864c1436-736d-4921-b137-ed7d41914a74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:59.244737 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.244654    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864c1436-736d-4921-b137-ed7d41914a74-kube-api-access-h6bkw" pod "864c1436-736d-4921-b137-ed7d41914a74" (UID: "864c1436-736d-4921-b137-ed7d41914a74"). InnerVolumeSpecName "kube-api-access-h6bkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:59.338109 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.338036    6030 generic.go:356] "Generic (PLEG): container finished" podID="864c1436-736d-4921-b137-ed7d41914a74" containerID="37777689b708c751660c389a9fc6aa23d69de38ccf066cd09d969fecfeb697e2" exitCode=0
Oct 30 20:01:59.338289 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.338117    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"
Oct 30 20:01:59.338289 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.338169    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725" event={"ID":"864c1436-736d-4921-b137-ed7d41914a74","Type":"ContainerDied","Data":"37777689b708c751660c389a9fc6aa23d69de38ccf066cd09d969fecfeb697e2"}
Oct 30 20:01:59.338289 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.338220    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725" event={"ID":"864c1436-736d-4921-b137-ed7d41914a74","Type":"ContainerDied","Data":"66d97f0c905397619edddc295ec80011ae2cb1cc7cbea01638720281c38b1652"}
Oct 30 20:01:59.338289 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.338237    6030 scope.go:122] "RemoveContainer" containerID="37777689b708c751660c389a9fc6aa23d69de38ccf066cd09d969fecfeb697e2"
Oct 30 20:01:59.343625 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.343601    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864c1436-736d-4921-b137-ed7d41914a74-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:59.343760 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.343628    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6bkw\" (UniqueName: \"kubernetes.io/projected/864c1436-736d-4921-b137-ed7d41914a74-kube-api-access-h6bkw\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:59.343760 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.343643    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/864c1436-736d-4921-b137-ed7d41914a74-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:59.349610 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.349594    6030 scope.go:122] "RemoveContainer" containerID="37777689b708c751660c389a9fc6aa23d69de38ccf066cd09d969fecfeb697e2"
Oct 30 20:01:59.349873 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:01:59.349854    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37777689b708c751660c389a9fc6aa23d69de38ccf066cd09d969fecfeb697e2\": container with ID starting with 37777689b708c751660c389a9fc6aa23d69de38ccf066cd09d969fecfeb697e2 not found: ID does not exist" containerID="37777689b708c751660c389a9fc6aa23d69de38ccf066cd09d969fecfeb697e2"
Oct 30 20:01:59.349927 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.349882    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37777689b708c751660c389a9fc6aa23d69de38ccf066cd09d969fecfeb697e2"} err="failed to get container status \"37777689b708c751660c389a9fc6aa23d69de38ccf066cd09d969fecfeb697e2\": rpc error: code = NotFound desc = could not find container \"37777689b708c751660c389a9fc6aa23d69de38ccf066cd09d969fecfeb697e2\": container with ID starting with 37777689b708c751660c389a9fc6aa23d69de38ccf066cd09d969fecfeb697e2 not found: ID does not exist"
Oct 30 20:01:59.400434 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.400404    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"]
Oct 30 20:01:59.404361 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:01:59.404337    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725"]
Oct 30 20:02:00.168476 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:00.168442    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-b24a1a/auth-test-model-b24a1a-kserve-5f8b7fffb5-52725" podUID="864c1436-736d-4921-b137-ed7d41914a74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.139:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:02:01.208263 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:01.208230    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="864c1436-736d-4921-b137-ed7d41914a74" path="/var/lib/kubelet/pods/864c1436-736d-4921-b137-ed7d41914a74/volumes"
Oct 30 20:02:04.324945 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:02:04.324905    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:02:06.057473 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:06.057440    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"]
Oct 30 20:02:06.058167 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:06.058151    6030 cpu_manager.go:414] "containerMap: removing container" podUID="864c1436-736d-4921-b137-ed7d41914a74" containerName="main"
Oct 30 20:02:06.058223 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:06.058168    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864c1436-736d-4921-b137-ed7d41914a74" containerName="main"
Oct 30 20:02:06.058281 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:06.058272    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="864c1436-736d-4921-b137-ed7d41914a74" containerName="main"
Oct 30 20:02:06.063525 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:06.063501    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"
Oct 30 20:02:06.079274 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:06.076622    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d97cc8-dockercfg-5ztc6\""
Oct 30 20:02:06.079274 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:06.078144    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d97cc8\""
Oct 30 20:02:06.081007 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:06.080940    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"]
Oct 30 20:02:06.108575 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:06.108539    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv8l7\" (UniqueName: \"kubernetes.io/projected/0b80bf01-a7f4-41ae-9adc-f550c6ec7707-kube-api-access-nv8l7\") pod \"payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc\" (UID: \"0b80bf01-a7f4-41ae-9adc-f550c6ec7707\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"
Oct 30 20:02:06.108786 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:06.108764    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b80bf01-a7f4-41ae-9adc-f550c6ec7707-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc\" (UID: \"0b80bf01-a7f4-41ae-9adc-f550c6ec7707\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"
Oct 30 20:02:06.219075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:06.217980    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv8l7\" (UniqueName: \"kubernetes.io/projected/0b80bf01-a7f4-41ae-9adc-f550c6ec7707-kube-api-access-nv8l7\") pod \"payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc\" (UID: \"0b80bf01-a7f4-41ae-9adc-f550c6ec7707\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"
Oct 30 20:02:06.219075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:06.218188    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b80bf01-a7f4-41ae-9adc-f550c6ec7707-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc\" (UID: \"0b80bf01-a7f4-41ae-9adc-f550c6ec7707\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"
Oct 30 20:02:06.234079 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:06.220010    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b80bf01-a7f4-41ae-9adc-f550c6ec7707-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc\" (UID: \"0b80bf01-a7f4-41ae-9adc-f550c6ec7707\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"
Oct 30 20:02:06.295815 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:06.295783    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv8l7\" (UniqueName: \"kubernetes.io/projected/0b80bf01-a7f4-41ae-9adc-f550c6ec7707-kube-api-access-nv8l7\") pod \"payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc\" (UID: \"0b80bf01-a7f4-41ae-9adc-f550c6ec7707\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"
Oct 30 20:02:06.332748 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:06.332661    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"
Oct 30 20:02:06.382863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:06.382832    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"
Oct 30 20:02:06.697026 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:06.696995    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"]
Oct 30 20:02:06.698242 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:02:06.698211    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b80bf01_a7f4_41ae_9adc_f550c6ec7707.slice/crio-b283e5499f2aefa577fdf11ef06e294eef9b38bcf720f847a10403297f380238 WatchSource:0}: Error finding container b283e5499f2aefa577fdf11ef06e294eef9b38bcf720f847a10403297f380238: Status 404 returned error can't find the container with id b283e5499f2aefa577fdf11ef06e294eef9b38bcf720f847a10403297f380238
Oct 30 20:02:07.471083 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:07.470650    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc" event={"ID":"0b80bf01-a7f4-41ae-9adc-f550c6ec7707","Type":"ContainerStarted","Data":"f73e85ae5b581eb4e54401dbdcb77d11b8667b94823158ccd159fd0d2062cf0a"}
Oct 30 20:02:07.471083 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:07.470718    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc" event={"ID":"0b80bf01-a7f4-41ae-9adc-f550c6ec7707","Type":"ContainerStarted","Data":"b283e5499f2aefa577fdf11ef06e294eef9b38bcf720f847a10403297f380238"}
Oct 30 20:02:07.475102 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:07.471878    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"
Oct 30 20:02:07.531424 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:07.531358    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc" podStartSLOduration=1.531337788 podStartE2EDuration="1.531337788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:07.51042314 +0000 UTC m=+1806.426030010" watchObservedRunningTime="2026-10-30 20:02:07.531337788 +0000 UTC m=+1806.446944645"
Oct 30 20:02:08.104253 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.104214    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"]
Oct 30 20:02:08.108666 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.108643    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.126235 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.126205    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"]
Oct 30 20:02:08.165747 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.165710    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-workload-certs\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.165953 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.165858    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-workload-socket\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.166027 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.165952    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-credential-socket\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.166027 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.166007    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5075d107-d1a3-4f56-afd0-6be45cbda01d-istiod-ca-cert\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.166167 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.166089    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-data\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.166167 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.166146    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-podinfo\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.166279 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.166265    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-envoy\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.166336 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.166291    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-token\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.166394 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.166383    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtqdx\" (UniqueName: \"kubernetes.io/projected/5075d107-d1a3-4f56-afd0-6be45cbda01d-kube-api-access-rtqdx\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.209897 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.209870    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-222480-openshift-default-dockercfg-2zjf9\""
Oct 30 20:02:08.267730 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.267694    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-workload-socket\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.267918 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.267735    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-credential-socket\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.267918 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.267784    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5075d107-d1a3-4f56-afd0-6be45cbda01d-istiod-ca-cert\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.267918 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.267838    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-data\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.267918 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.267888    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-podinfo\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.268221 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.268008    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-envoy\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.268221 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.268033    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-token\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.268221 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.268160    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtqdx\" (UniqueName: \"kubernetes.io/projected/5075d107-d1a3-4f56-afd0-6be45cbda01d-kube-api-access-rtqdx\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.268385 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.268231    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-workload-certs\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.268385 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.268250    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-credential-socket\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.268385 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.268267    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-data\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.268668 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.268644    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-workload-certs\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.268824 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.268695    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5075d107-d1a3-4f56-afd0-6be45cbda01d-istiod-ca-cert\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.268873 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.268850    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-workload-socket\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.271110 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.271080    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-envoy\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.271274 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.271259    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-podinfo\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.283826 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.283802    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-token\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.287059 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.287020    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtqdx\" (UniqueName: \"kubernetes.io/projected/5075d107-d1a3-4f56-afd0-6be45cbda01d-kube-api-access-rtqdx\") pod \"e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") " pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:08.438542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:08.437004    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:09.193215 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:09.193183    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"]
Oct 30 20:02:09.193967 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:02:09.193938    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5075d107_d1a3_4f56_afd0_6be45cbda01d.slice/crio-0fcb4aebe4d8118afabed93b1e41f22ee71a011b533486765f0bb60e20b5c8d1 WatchSource:0}: Error finding container 0fcb4aebe4d8118afabed93b1e41f22ee71a011b533486765f0bb60e20b5c8d1: Status 404 returned error can't find the container with id 0fcb4aebe4d8118afabed93b1e41f22ee71a011b533486765f0bb60e20b5c8d1
Oct 30 20:02:09.197443 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:09.197410    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:02:09.197601 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:09.197475    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:02:09.197601 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:09.197510    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:02:09.518105 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:09.517990    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8" event={"ID":"5075d107-d1a3-4f56-afd0-6be45cbda01d","Type":"ContainerStarted","Data":"28d5fb61c8d48c4c29739e388eee1b80641ecc70845eea1e282c0ff04af5b158"}
Oct 30 20:02:09.518287 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:09.518129    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8" event={"ID":"5075d107-d1a3-4f56-afd0-6be45cbda01d","Type":"ContainerStarted","Data":"0fcb4aebe4d8118afabed93b1e41f22ee71a011b533486765f0bb60e20b5c8d1"}
Oct 30 20:02:09.555553 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:09.555492    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8" podStartSLOduration=1.55547379 podStartE2EDuration="1.55547379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:09.552199234 +0000 UTC m=+1808.467806091" watchObservedRunningTime="2026-10-30 20:02:09.55547379 +0000 UTC m=+1808.471080648"
Oct 30 20:02:10.437651 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:10.437617    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:10.443587 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:10.443559    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:10.524509 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:10.524467    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:10.525426 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:10.525402    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:02:11.379033 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:11.379002    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"]
Oct 30 20:02:11.383302 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:11.383281    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:02:11.393881 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:11.393845    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-8c8241-serving-cert\""
Oct 30 20:02:11.398804 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:11.398770    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"]
Oct 30 20:02:11.408671 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:11.408645    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmhwn\" (UniqueName: \"kubernetes.io/projected/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-kube-api-access-qmhwn\") pod \"maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477\" (UID: \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:02:11.408794 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:11.408705    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477\" (UID: \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:02:11.408871 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:11.408845    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-metrics-certs\") pod \"maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477\" (UID: \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:02:11.409025 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:11.409008    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477\" (UID: \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:02:11.510560 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:11.510525    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477\" (UID: \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:02:11.511031 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:11.510648    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhwn\" (UniqueName: \"kubernetes.io/projected/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-kube-api-access-qmhwn\") pod \"maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477\" (UID: \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:02:11.511031 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:11.510690    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477\" (UID: \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:02:11.511031 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:11.510751    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-metrics-certs\") pod \"maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477\" (UID: \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:02:11.511031 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:02:11.510893    6030 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-serving-cert: secret "maas-api-e2e-extmod-a-8c8241-serving-cert" not found
Oct 30 20:02:11.511031 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:02:11.510971    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-maas-api-tls podName:c3a85e15-16f9-4e1f-aa08-f1437a0bb44f nodeName:}" failed. No retries permitted until 2026-10-30 20:02:12.010949885 +0000 UTC m=+1810.926556736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-maas-api-tls") pod "maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477" (UID: "c3a85e15-16f9-4e1f-aa08-f1437a0bb44f") : secret "maas-api-e2e-extmod-a-8c8241-serving-cert" not found
Oct 30 20:02:11.511398 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:11.511380    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477\" (UID: \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:02:11.513504 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:11.513486    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-metrics-certs\") pod \"maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477\" (UID: \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:02:11.526060 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:11.526014    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmhwn\" (UniqueName: \"kubernetes.io/projected/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-kube-api-access-qmhwn\") pod \"maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477\" (UID: \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:02:12.017163 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:12.017132    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477\" (UID: \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:02:12.019894 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:12.019868    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477\" (UID: \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:02:12.302104 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:12.302018    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:02:12.685408 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:12.683596    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"]
Oct 30 20:02:13.574896 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:13.574805    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477" event={"ID":"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f","Type":"ContainerStarted","Data":"c631125ab62e34e8bf2d233d0f14d8ae6a176086b05b749fb036762ff94c4832"}
Oct 30 20:02:13.575080 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:13.574896    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477" event={"ID":"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f","Type":"ContainerStarted","Data":"09fecd3360db7ba271fb17fc6b3941fae29696f258c8829b8b38c7dfdc3a2df3"}
Oct 30 20:02:13.575176 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:13.575072    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:02:13.604990 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:13.604941    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477" podStartSLOduration=2.076893004 podStartE2EDuration="2.604927205s" totalImagesPullingTime="528.034201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:13.604353319 +0000 UTC m=+1812.519960175" watchObservedRunningTime="2026-10-30 20:02:13.604927205 +0000 UTC m=+1812.520534060"
Oct 30 20:02:14.006071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:13.974034    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"]
Oct 30 20:02:18.332625 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:18.332582    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"]
Oct 30 20:02:18.985647 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:18.985611    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"]
Oct 30 20:02:18.985906 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:18.985870    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz" podUID="2c441eb5-f8ed-413a-b77c-4a8e043c431e" containerName="istio-proxy" containerID="cri-o://c3f716db95a5b245a94ac8f5edf371a43b278233b1931af4ebdbe6b64bdafac3" gracePeriod=30
Oct 30 20:02:18.991122 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:18.991094    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"]
Oct 30 20:02:18.991337 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:18.991303    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf" podUID="70cb2df1-ae6b-49d3-a989-db43421d61d6" containerName="istio-proxy" containerID="cri-o://ee9f01e23e5449306590c724d6eefe9bbf0c2c7849b56e70130d640b0cea6113" gracePeriod=30
Oct 30 20:02:18.991981 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:18.991961    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"]
Oct 30 20:02:18.992204 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:18.992163    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2" podUID="a7b54a5e-b3af-466f-990e-21e6b60ae71b" containerName="istio-proxy" containerID="cri-o://859362de05e709033c670d35920b9610b6533473cefb1fe7d685988a2345d19e" gracePeriod=30
Oct 30 20:02:19.522425 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:19.522396    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"
Oct 30 20:02:19.854982 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:19.852679    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:02:24.348488 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.348448    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 20:02:24.416369 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.416340    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 20:02:24.419329 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.419311    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 20:02:24.457106 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.457074    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-envoy\") pod \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") "
Oct 30 20:02:24.457289 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.457180    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-workload-certs\") pod \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") "
Oct 30 20:02:24.457289 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.457211    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-workload-socket\") pod \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") "
Oct 30 20:02:24.457289 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.457232    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istiod-ca-cert\") pod \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") "
Oct 30 20:02:24.457472 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.457321    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-credential-socket\") pod \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") "
Oct 30 20:02:24.457472 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.457353    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-token\" (UniqueName: \"kubernetes.io/projected/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-token\") pod \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") "
Oct 30 20:02:24.457472 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.457397    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c441eb5-f8ed-413a-b77c-4a8e043c431e-kube-api-access-k6c8x\" (UniqueName: \"kubernetes.io/projected/2c441eb5-f8ed-413a-b77c-4a8e043c431e-kube-api-access-k6c8x\") pod \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") "
Oct 30 20:02:24.457472 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.457427    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-data\") pod \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") "
Oct 30 20:02:24.457679 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.457549    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-podinfo\") pod \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\" (UID: \"2c441eb5-f8ed-413a-b77c-4a8e043c431e\") "
Oct 30 20:02:24.458496 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.457546    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-workload-certs" pod "2c441eb5-f8ed-413a-b77c-4a8e043c431e" (UID: "2c441eb5-f8ed-413a-b77c-4a8e043c431e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.458619 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.458215    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-data" pod "2c441eb5-f8ed-413a-b77c-4a8e043c431e" (UID: "2c441eb5-f8ed-413a-b77c-4a8e043c431e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.458619 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.458422    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-workload-socket" pod "2c441eb5-f8ed-413a-b77c-4a8e043c431e" (UID: "2c441eb5-f8ed-413a-b77c-4a8e043c431e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.458797 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.458749    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-credential-socket" pod "2c441eb5-f8ed-413a-b77c-4a8e043c431e" (UID: "2c441eb5-f8ed-413a-b77c-4a8e043c431e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.458967 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.458920    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istiod-ca-cert" pod "2c441eb5-f8ed-413a-b77c-4a8e043c431e" (UID: "2c441eb5-f8ed-413a-b77c-4a8e043c431e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:24.460526 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.460496    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-token" pod "2c441eb5-f8ed-413a-b77c-4a8e043c431e" (UID: "2c441eb5-f8ed-413a-b77c-4a8e043c431e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:24.460879 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.460851    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-podinfo" pod "2c441eb5-f8ed-413a-b77c-4a8e043c431e" (UID: "2c441eb5-f8ed-413a-b77c-4a8e043c431e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:24.461089 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.461067    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-envoy" pod "2c441eb5-f8ed-413a-b77c-4a8e043c431e" (UID: "2c441eb5-f8ed-413a-b77c-4a8e043c431e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.461443 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.461422    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c441eb5-f8ed-413a-b77c-4a8e043c431e-kube-api-access-k6c8x" pod "2c441eb5-f8ed-413a-b77c-4a8e043c431e" (UID: "2c441eb5-f8ed-413a-b77c-4a8e043c431e"). InnerVolumeSpecName "kube-api-access-k6c8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:24.559266 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.559177    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-workload-socket\") pod \"70cb2df1-ae6b-49d3-a989-db43421d61d6\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") "
Oct 30 20:02:24.559266 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.559220    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-podinfo\") pod \"70cb2df1-ae6b-49d3-a989-db43421d61d6\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") "
Oct 30 20:02:24.559266 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.559251    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70cb2df1-ae6b-49d3-a989-db43421d61d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70cb2df1-ae6b-49d3-a989-db43421d61d6-istiod-ca-cert\") pod \"70cb2df1-ae6b-49d3-a989-db43421d61d6\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") "
Oct 30 20:02:24.559534 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.559292    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-workload-socket\") pod \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") "
Oct 30 20:02:24.559534 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.559347    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-envoy\") pod \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") "
Oct 30 20:02:24.559534 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.559395    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-podinfo\") pod \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") "
Oct 30 20:02:24.559534 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.559425    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-data\") pod \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") "
Oct 30 20:02:24.559534 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.559470    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-workload-socket" pod "70cb2df1-ae6b-49d3-a989-db43421d61d6" (UID: "70cb2df1-ae6b-49d3-a989-db43421d61d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.559534 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.559511    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7b54a5e-b3af-466f-990e-21e6b60ae71b-kube-api-access-b7vms\" (UniqueName: \"kubernetes.io/projected/a7b54a5e-b3af-466f-990e-21e6b60ae71b-kube-api-access-b7vms\") pod \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") "
Oct 30 20:02:24.559901 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.559562    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-credential-socket\") pod \"70cb2df1-ae6b-49d3-a989-db43421d61d6\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") "
Oct 30 20:02:24.559901 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.559648    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-workload-certs\") pod \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") "
Oct 30 20:02:24.559901 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.559686    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-workload-socket" pod "a7b54a5e-b3af-466f-990e-21e6b60ae71b" (UID: "a7b54a5e-b3af-466f-990e-21e6b60ae71b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.559901 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.559708    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-data\") pod \"70cb2df1-ae6b-49d3-a989-db43421d61d6\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") "
Oct 30 20:02:24.559901 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.559793    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-token\" (UniqueName: \"kubernetes.io/projected/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-token\") pod \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") "
Oct 30 20:02:24.559901 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.559832    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70cb2df1-ae6b-49d3-a989-db43421d61d6-kube-api-access-bbqwl\" (UniqueName: \"kubernetes.io/projected/70cb2df1-ae6b-49d3-a989-db43421d61d6-kube-api-access-bbqwl\") pod \"70cb2df1-ae6b-49d3-a989-db43421d61d6\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") "
Oct 30 20:02:24.559901 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.559876    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-workload-certs\") pod \"70cb2df1-ae6b-49d3-a989-db43421d61d6\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") "
Oct 30 20:02:24.560416 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.559958    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istiod-ca-cert\") pod \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") "
Oct 30 20:02:24.560416 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.560028    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-envoy\") pod \"70cb2df1-ae6b-49d3-a989-db43421d61d6\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") "
Oct 30 20:02:24.560416 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.560095    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-credential-socket\") pod \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\" (UID: \"a7b54a5e-b3af-466f-990e-21e6b60ae71b\") "
Oct 30 20:02:24.560416 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.560152    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-token\" (UniqueName: \"kubernetes.io/projected/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-token\") pod \"70cb2df1-ae6b-49d3-a989-db43421d61d6\" (UID: \"70cb2df1-ae6b-49d3-a989-db43421d61d6\") "
Oct 30 20:02:24.560605 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.560541    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-workload-certs" pod "a7b54a5e-b3af-466f-990e-21e6b60ae71b" (UID: "a7b54a5e-b3af-466f-990e-21e6b60ae71b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.562142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.560891    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70cb2df1-ae6b-49d3-a989-db43421d61d6-istiod-ca-cert" pod "70cb2df1-ae6b-49d3-a989-db43421d61d6" (UID: "70cb2df1-ae6b-49d3-a989-db43421d61d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:24.562142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.560903    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.562142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.560942    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.562142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.560959    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.562142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.560973    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.562142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.560987    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.562142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.561002    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6c8x\" (UniqueName: \"kubernetes.io/projected/2c441eb5-f8ed-413a-b77c-4a8e043c431e-kube-api-access-k6c8x\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.562142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.561016    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.562142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.561030    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.562142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.561164    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.562142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.561180    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.562142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.561193    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c441eb5-f8ed-413a-b77c-4a8e043c431e-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.562142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.561205    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.562142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.561142    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-credential-socket" pod "70cb2df1-ae6b-49d3-a989-db43421d61d6" (UID: "70cb2df1-ae6b-49d3-a989-db43421d61d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.562142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.561512    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istiod-ca-cert" pod "a7b54a5e-b3af-466f-990e-21e6b60ae71b" (UID: "a7b54a5e-b3af-466f-990e-21e6b60ae71b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:24.562142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.561797    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-data" pod "70cb2df1-ae6b-49d3-a989-db43421d61d6" (UID: "70cb2df1-ae6b-49d3-a989-db43421d61d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.562142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.562026    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-workload-certs" pod "70cb2df1-ae6b-49d3-a989-db43421d61d6" (UID: "70cb2df1-ae6b-49d3-a989-db43421d61d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.563334 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.562212    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-credential-socket" pod "a7b54a5e-b3af-466f-990e-21e6b60ae71b" (UID: "a7b54a5e-b3af-466f-990e-21e6b60ae71b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.563334 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.563279    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-data" pod "a7b54a5e-b3af-466f-990e-21e6b60ae71b" (UID: "a7b54a5e-b3af-466f-990e-21e6b60ae71b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.563679 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.563646    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70cb2df1-ae6b-49d3-a989-db43421d61d6-kube-api-access-bbqwl" pod "70cb2df1-ae6b-49d3-a989-db43421d61d6" (UID: "70cb2df1-ae6b-49d3-a989-db43421d61d6"). InnerVolumeSpecName "kube-api-access-bbqwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:24.564457 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.564433    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-envoy" pod "70cb2df1-ae6b-49d3-a989-db43421d61d6" (UID: "70cb2df1-ae6b-49d3-a989-db43421d61d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.565586 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.565550    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-podinfo" pod "70cb2df1-ae6b-49d3-a989-db43421d61d6" (UID: "70cb2df1-ae6b-49d3-a989-db43421d61d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:24.565872 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.565849    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-token" pod "a7b54a5e-b3af-466f-990e-21e6b60ae71b" (UID: "a7b54a5e-b3af-466f-990e-21e6b60ae71b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:24.565989 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.565966    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-podinfo" pod "a7b54a5e-b3af-466f-990e-21e6b60ae71b" (UID: "a7b54a5e-b3af-466f-990e-21e6b60ae71b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:24.566095 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.565969    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b54a5e-b3af-466f-990e-21e6b60ae71b-kube-api-access-b7vms" pod "a7b54a5e-b3af-466f-990e-21e6b60ae71b" (UID: "a7b54a5e-b3af-466f-990e-21e6b60ae71b"). InnerVolumeSpecName "kube-api-access-b7vms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:24.566265 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.566248    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-envoy" pod "a7b54a5e-b3af-466f-990e-21e6b60ae71b" (UID: "a7b54a5e-b3af-466f-990e-21e6b60ae71b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.566870 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.566848    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-token" pod "70cb2df1-ae6b-49d3-a989-db43421d61d6" (UID: "70cb2df1-ae6b-49d3-a989-db43421d61d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:24.662447 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.662411    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.662447 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.662445    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.662641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.662457    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70cb2df1-ae6b-49d3-a989-db43421d61d6-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.662641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.662466    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.662641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.662475    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.662641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.662497    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.662641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.662507    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7vms\" (UniqueName: \"kubernetes.io/projected/a7b54a5e-b3af-466f-990e-21e6b60ae71b-kube-api-access-b7vms\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.662641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.662517    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.662641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.662525    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.662641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.662534    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.662641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.662543    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbqwl\" (UniqueName: \"kubernetes.io/projected/70cb2df1-ae6b-49d3-a989-db43421d61d6-kube-api-access-bbqwl\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.662641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.662551    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.662641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.662559    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7b54a5e-b3af-466f-990e-21e6b60ae71b-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.662641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.662567    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70cb2df1-ae6b-49d3-a989-db43421d61d6-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.662641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.662575    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b54a5e-b3af-466f-990e-21e6b60ae71b-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.726579 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.726545    6030 generic.go:356] "Generic (PLEG): container finished" podID="2c441eb5-f8ed-413a-b77c-4a8e043c431e" containerID="c3f716db95a5b245a94ac8f5edf371a43b278233b1931af4ebdbe6b64bdafac3" exitCode=0
Oct 30 20:02:24.726762 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.726599    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"
Oct 30 20:02:24.726762 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.726620    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz" event={"ID":"2c441eb5-f8ed-413a-b77c-4a8e043c431e","Type":"ContainerDied","Data":"c3f716db95a5b245a94ac8f5edf371a43b278233b1931af4ebdbe6b64bdafac3"}
Oct 30 20:02:24.726762 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.726692    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz" event={"ID":"2c441eb5-f8ed-413a-b77c-4a8e043c431e","Type":"ContainerDied","Data":"55938ff5d133fc722e43d4fb6cf99261222b46983bc00600c883fa6f1b6bb28d"}
Oct 30 20:02:24.726762 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.726721    6030 scope.go:122] "RemoveContainer" containerID="c3f716db95a5b245a94ac8f5edf371a43b278233b1931af4ebdbe6b64bdafac3"
Oct 30 20:02:24.728291 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.728270    6030 generic.go:356] "Generic (PLEG): container finished" podID="a7b54a5e-b3af-466f-990e-21e6b60ae71b" containerID="859362de05e709033c670d35920b9610b6533473cefb1fe7d685988a2345d19e" exitCode=0
Oct 30 20:02:24.728486 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.728313    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"
Oct 30 20:02:24.728486 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.728308    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2" event={"ID":"a7b54a5e-b3af-466f-990e-21e6b60ae71b","Type":"ContainerDied","Data":"859362de05e709033c670d35920b9610b6533473cefb1fe7d685988a2345d19e"}
Oct 30 20:02:24.728486 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.728444    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2" event={"ID":"a7b54a5e-b3af-466f-990e-21e6b60ae71b","Type":"ContainerDied","Data":"630d82043e5465ef6163b3328c9dbccf689b53f943ebe79336bfe519242017e3"}
Oct 30 20:02:24.732773 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.732729    6030 generic.go:356] "Generic (PLEG): container finished" podID="70cb2df1-ae6b-49d3-a989-db43421d61d6" containerID="ee9f01e23e5449306590c724d6eefe9bbf0c2c7849b56e70130d640b0cea6113" exitCode=0
Oct 30 20:02:24.732959 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.732802    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"
Oct 30 20:02:24.732959 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.732863    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf" event={"ID":"70cb2df1-ae6b-49d3-a989-db43421d61d6","Type":"ContainerDied","Data":"ee9f01e23e5449306590c724d6eefe9bbf0c2c7849b56e70130d640b0cea6113"}
Oct 30 20:02:24.732959 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.732899    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf" event={"ID":"70cb2df1-ae6b-49d3-a989-db43421d61d6","Type":"ContainerDied","Data":"7aebe46041acbcd15dac2c2ad79d0e77d744abe9e5d992388284dc441aaf543d"}
Oct 30 20:02:24.748380 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.748357    6030 scope.go:122] "RemoveContainer" containerID="c3f716db95a5b245a94ac8f5edf371a43b278233b1931af4ebdbe6b64bdafac3"
Oct 30 20:02:24.748788 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:02:24.748765    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3f716db95a5b245a94ac8f5edf371a43b278233b1931af4ebdbe6b64bdafac3\": container with ID starting with c3f716db95a5b245a94ac8f5edf371a43b278233b1931af4ebdbe6b64bdafac3 not found: ID does not exist" containerID="c3f716db95a5b245a94ac8f5edf371a43b278233b1931af4ebdbe6b64bdafac3"
Oct 30 20:02:24.748876 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.748797    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3f716db95a5b245a94ac8f5edf371a43b278233b1931af4ebdbe6b64bdafac3"} err="failed to get container status \"c3f716db95a5b245a94ac8f5edf371a43b278233b1931af4ebdbe6b64bdafac3\": rpc error: code = NotFound desc = could not find container \"c3f716db95a5b245a94ac8f5edf371a43b278233b1931af4ebdbe6b64bdafac3\": container with ID starting with c3f716db95a5b245a94ac8f5edf371a43b278233b1931af4ebdbe6b64bdafac3 not found: ID does not exist"
Oct 30 20:02:24.748876 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.748816    6030 scope.go:122] "RemoveContainer" containerID="859362de05e709033c670d35920b9610b6533473cefb1fe7d685988a2345d19e"
Oct 30 20:02:24.773210 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.773178    6030 scope.go:122] "RemoveContainer" containerID="859362de05e709033c670d35920b9610b6533473cefb1fe7d685988a2345d19e"
Oct 30 20:02:24.773582 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:02:24.773557    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"859362de05e709033c670d35920b9610b6533473cefb1fe7d685988a2345d19e\": container with ID starting with 859362de05e709033c670d35920b9610b6533473cefb1fe7d685988a2345d19e not found: ID does not exist" containerID="859362de05e709033c670d35920b9610b6533473cefb1fe7d685988a2345d19e"
Oct 30 20:02:24.773693 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.773588    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859362de05e709033c670d35920b9610b6533473cefb1fe7d685988a2345d19e"} err="failed to get container status \"859362de05e709033c670d35920b9610b6533473cefb1fe7d685988a2345d19e\": rpc error: code = NotFound desc = could not find container \"859362de05e709033c670d35920b9610b6533473cefb1fe7d685988a2345d19e\": container with ID starting with 859362de05e709033c670d35920b9610b6533473cefb1fe7d685988a2345d19e not found: ID does not exist"
Oct 30 20:02:24.773693 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.773605    6030 scope.go:122] "RemoveContainer" containerID="ee9f01e23e5449306590c724d6eefe9bbf0c2c7849b56e70130d640b0cea6113"
Oct 30 20:02:24.784316 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.784294    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"]
Oct 30 20:02:24.784882 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.784854    6030 scope.go:122] "RemoveContainer" containerID="ee9f01e23e5449306590c724d6eefe9bbf0c2c7849b56e70130d640b0cea6113"
Oct 30 20:02:24.787541 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:02:24.787516    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9f01e23e5449306590c724d6eefe9bbf0c2c7849b56e70130d640b0cea6113\": container with ID starting with ee9f01e23e5449306590c724d6eefe9bbf0c2c7849b56e70130d640b0cea6113 not found: ID does not exist" containerID="ee9f01e23e5449306590c724d6eefe9bbf0c2c7849b56e70130d640b0cea6113"
Oct 30 20:02:24.787914 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.787549    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9f01e23e5449306590c724d6eefe9bbf0c2c7849b56e70130d640b0cea6113"} err="failed to get container status \"ee9f01e23e5449306590c724d6eefe9bbf0c2c7849b56e70130d640b0cea6113\": rpc error: code = NotFound desc = could not find container \"ee9f01e23e5449306590c724d6eefe9bbf0c2c7849b56e70130d640b0cea6113\": container with ID starting with ee9f01e23e5449306590c724d6eefe9bbf0c2c7849b56e70130d640b0cea6113 not found: ID does not exist"
Oct 30 20:02:24.793868 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.793816    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-8rltz"]
Oct 30 20:02:24.819215 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.819144    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"]
Oct 30 20:02:24.828116 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.828084    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-v2dw2"]
Oct 30 20:02:24.841542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.841517    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"]
Oct 30 20:02:24.848367 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.848340    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-hwvnf"]
Oct 30 20:02:24.937530 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.937499    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"]
Oct 30 20:02:24.938121 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.938107    6030 cpu_manager.go:414] "containerMap: removing container" podUID="70cb2df1-ae6b-49d3-a989-db43421d61d6" containerName="istio-proxy"
Oct 30 20:02:24.938193 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.938121    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70cb2df1-ae6b-49d3-a989-db43421d61d6" containerName="istio-proxy"
Oct 30 20:02:24.938193 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.938150    6030 cpu_manager.go:414] "containerMap: removing container" podUID="a7b54a5e-b3af-466f-990e-21e6b60ae71b" containerName="istio-proxy"
Oct 30 20:02:24.938193 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.938155    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7b54a5e-b3af-466f-990e-21e6b60ae71b" containerName="istio-proxy"
Oct 30 20:02:24.938193 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.938173    6030 cpu_manager.go:414] "containerMap: removing container" podUID="2c441eb5-f8ed-413a-b77c-4a8e043c431e" containerName="istio-proxy"
Oct 30 20:02:24.938193 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.938177    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c441eb5-f8ed-413a-b77c-4a8e043c431e" containerName="istio-proxy"
Oct 30 20:02:24.938345 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.938272    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="70cb2df1-ae6b-49d3-a989-db43421d61d6" containerName="istio-proxy"
Oct 30 20:02:24.938345 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.938285    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7b54a5e-b3af-466f-990e-21e6b60ae71b" containerName="istio-proxy"
Oct 30 20:02:24.938345 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.938293    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c441eb5-f8ed-413a-b77c-4a8e043c431e" containerName="istio-proxy"
Oct 30 20:02:24.944022 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.943998    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"
Oct 30 20:02:24.949690 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.949667    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-836d67\""
Oct 30 20:02:24.956066 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.956021    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"]
Oct 30 20:02:24.974113 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.974079    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"]
Oct 30 20:02:24.978460 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.978440    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"
Oct 30 20:02:24.984563 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:24.984529    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-836d67-dockercfg-wsxlf\""
Oct 30 20:02:25.001965 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.001939    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"]
Oct 30 20:02:25.092129 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.089832    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1682349-9425-4aba-a4b6-01bbe7427c61-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8\" (UID: \"e1682349-9425-4aba-a4b6-01bbe7427c61\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"
Oct 30 20:02:25.092129 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.089992    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zptk\" (UniqueName: \"kubernetes.io/projected/73882a2e-adf8-4aaa-9da6-49d5e8917248-kube-api-access-7zptk\") pod \"payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx\" (UID: \"73882a2e-adf8-4aaa-9da6-49d5e8917248\") " pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"
Oct 30 20:02:25.092129 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.090031    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb7f9\" (UniqueName: \"kubernetes.io/projected/e1682349-9425-4aba-a4b6-01bbe7427c61-kube-api-access-pb7f9\") pod \"payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8\" (UID: \"e1682349-9425-4aba-a4b6-01bbe7427c61\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"
Oct 30 20:02:25.092129 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.090081    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73882a2e-adf8-4aaa-9da6-49d5e8917248-plugins-config-volume\") pod \"payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx\" (UID: \"73882a2e-adf8-4aaa-9da6-49d5e8917248\") " pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"
Oct 30 20:02:25.195070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.191786    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zptk\" (UniqueName: \"kubernetes.io/projected/73882a2e-adf8-4aaa-9da6-49d5e8917248-kube-api-access-7zptk\") pod \"payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx\" (UID: \"73882a2e-adf8-4aaa-9da6-49d5e8917248\") " pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"
Oct 30 20:02:25.195070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.191837    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb7f9\" (UniqueName: \"kubernetes.io/projected/e1682349-9425-4aba-a4b6-01bbe7427c61-kube-api-access-pb7f9\") pod \"payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8\" (UID: \"e1682349-9425-4aba-a4b6-01bbe7427c61\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"
Oct 30 20:02:25.195070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.191872    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73882a2e-adf8-4aaa-9da6-49d5e8917248-plugins-config-volume\") pod \"payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx\" (UID: \"73882a2e-adf8-4aaa-9da6-49d5e8917248\") " pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"
Oct 30 20:02:25.195070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.192100    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1682349-9425-4aba-a4b6-01bbe7427c61-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8\" (UID: \"e1682349-9425-4aba-a4b6-01bbe7427c61\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"
Oct 30 20:02:25.195070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.193032    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1682349-9425-4aba-a4b6-01bbe7427c61-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8\" (UID: \"e1682349-9425-4aba-a4b6-01bbe7427c61\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"
Oct 30 20:02:25.209196 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.209128    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73882a2e-adf8-4aaa-9da6-49d5e8917248-plugins-config-volume\") pod \"payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx\" (UID: \"73882a2e-adf8-4aaa-9da6-49d5e8917248\") " pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"
Oct 30 20:02:25.210803 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.210722    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zptk\" (UniqueName: \"kubernetes.io/projected/73882a2e-adf8-4aaa-9da6-49d5e8917248-kube-api-access-7zptk\") pod \"payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx\" (UID: \"73882a2e-adf8-4aaa-9da6-49d5e8917248\") " pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"
Oct 30 20:02:25.241696 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.239345    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb7f9\" (UniqueName: \"kubernetes.io/projected/e1682349-9425-4aba-a4b6-01bbe7427c61-kube-api-access-pb7f9\") pod \"payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8\" (UID: \"e1682349-9425-4aba-a4b6-01bbe7427c61\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"
Oct 30 20:02:25.255099 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.254298    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c441eb5-f8ed-413a-b77c-4a8e043c431e" path="/var/lib/kubelet/pods/2c441eb5-f8ed-413a-b77c-4a8e043c431e/volumes"
Oct 30 20:02:25.255099 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.254957    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70cb2df1-ae6b-49d3-a989-db43421d61d6" path="/var/lib/kubelet/pods/70cb2df1-ae6b-49d3-a989-db43421d61d6/volumes"
Oct 30 20:02:25.256916 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.256887    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7b54a5e-b3af-466f-990e-21e6b60ae71b" path="/var/lib/kubelet/pods/a7b54a5e-b3af-466f-990e-21e6b60ae71b/volumes"
Oct 30 20:02:25.260754 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.260716    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"
Oct 30 20:02:25.306448 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.306412    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"
Oct 30 20:02:25.593773 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.593736    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"]
Oct 30 20:02:25.627923 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:02:25.627894    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73882a2e_adf8_4aaa_9da6_49d5e8917248.slice/crio-afdd17f4e0dc56a2927e99a3d0fb9bc71ee95041a563c1b311cc3f657f1c008f WatchSource:0}: Error finding container afdd17f4e0dc56a2927e99a3d0fb9bc71ee95041a563c1b311cc3f657f1c008f: Status 404 returned error can't find the container with id afdd17f4e0dc56a2927e99a3d0fb9bc71ee95041a563c1b311cc3f657f1c008f
Oct 30 20:02:25.628803 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.628776    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"]
Oct 30 20:02:25.740604 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.740570    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx" event={"ID":"73882a2e-adf8-4aaa-9da6-49d5e8917248","Type":"ContainerStarted","Data":"ba6c94bef6f2ab4f5c3d43645a0f437429d6c2f355811a609d776630e41de320"}
Oct 30 20:02:25.740752 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.740640    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"
Oct 30 20:02:25.740752 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.740668    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx" event={"ID":"73882a2e-adf8-4aaa-9da6-49d5e8917248","Type":"ContainerStarted","Data":"afdd17f4e0dc56a2927e99a3d0fb9bc71ee95041a563c1b311cc3f657f1c008f"}
Oct 30 20:02:25.744226 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.744193    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8" event={"ID":"e1682349-9425-4aba-a4b6-01bbe7427c61","Type":"ContainerStarted","Data":"06f4bcfb01a968c81ddf60caf25a3c68e8c25600173a87970b23c1b13c3c1936"}
Oct 30 20:02:25.744349 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.744240    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"
Oct 30 20:02:25.744349 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.744263    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8" event={"ID":"e1682349-9425-4aba-a4b6-01bbe7427c61","Type":"ContainerStarted","Data":"ed642b5de27721562b2a88ad021deef415e5460ceca9b38299352e4371dfc887"}
Oct 30 20:02:25.773285 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.773226    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx" podStartSLOduration=1.7732092210000001 podStartE2EDuration="1.773209221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:25.771802488 +0000 UTC m=+1824.687409348" watchObservedRunningTime="2026-10-30 20:02:25.773209221 +0000 UTC m=+1824.688816080"
Oct 30 20:02:25.801774 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:25.801726    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8" podStartSLOduration=1.80171056 podStartE2EDuration="1.80171056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:25.799000064 +0000 UTC m=+1824.714606921" watchObservedRunningTime="2026-10-30 20:02:25.80171056 +0000 UTC m=+1824.717317432"
Oct 30 20:02:26.730668 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:26.730635    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"]
Oct 30 20:02:26.736221 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:26.736195    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:02:26.749753 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:26.749711    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"]
Oct 30 20:02:26.798714 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:26.798683    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-d97cc8-serving-cert\""
Oct 30 20:02:26.928677 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:26.928640    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef621cd9-9b74-457a-a134-472f7a7b89ca-metrics-certs\") pod \"maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz\" (UID: \"ef621cd9-9b74-457a-a134-472f7a7b89ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:02:26.928849 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:26.928780    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbhnf\" (UniqueName: \"kubernetes.io/projected/ef621cd9-9b74-457a-a134-472f7a7b89ca-kube-api-access-bbhnf\") pod \"maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz\" (UID: \"ef621cd9-9b74-457a-a134-472f7a7b89ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:02:26.929013 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:26.928987    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef621cd9-9b74-457a-a134-472f7a7b89ca-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz\" (UID: \"ef621cd9-9b74-457a-a134-472f7a7b89ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:02:26.929184 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:26.929167    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef621cd9-9b74-457a-a134-472f7a7b89ca-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz\" (UID: \"ef621cd9-9b74-457a-a134-472f7a7b89ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:02:27.030568 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:27.030486    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef621cd9-9b74-457a-a134-472f7a7b89ca-metrics-certs\") pod \"maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz\" (UID: \"ef621cd9-9b74-457a-a134-472f7a7b89ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:02:27.030568 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:27.030546    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbhnf\" (UniqueName: \"kubernetes.io/projected/ef621cd9-9b74-457a-a134-472f7a7b89ca-kube-api-access-bbhnf\") pod \"maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz\" (UID: \"ef621cd9-9b74-457a-a134-472f7a7b89ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:02:27.030803 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:27.030619    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef621cd9-9b74-457a-a134-472f7a7b89ca-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz\" (UID: \"ef621cd9-9b74-457a-a134-472f7a7b89ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:02:27.030803 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:27.030705    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef621cd9-9b74-457a-a134-472f7a7b89ca-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz\" (UID: \"ef621cd9-9b74-457a-a134-472f7a7b89ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:02:27.031392 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:27.031366    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef621cd9-9b74-457a-a134-472f7a7b89ca-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz\" (UID: \"ef621cd9-9b74-457a-a134-472f7a7b89ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:02:27.033247 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:27.033223    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef621cd9-9b74-457a-a134-472f7a7b89ca-metrics-certs\") pod \"maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz\" (UID: \"ef621cd9-9b74-457a-a134-472f7a7b89ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:02:27.033247 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:27.033239    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef621cd9-9b74-457a-a134-472f7a7b89ca-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz\" (UID: \"ef621cd9-9b74-457a-a134-472f7a7b89ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:02:27.046796 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:27.046764    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbhnf\" (UniqueName: \"kubernetes.io/projected/ef621cd9-9b74-457a-a134-472f7a7b89ca-kube-api-access-bbhnf\") pod \"maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz\" (UID: \"ef621cd9-9b74-457a-a134-472f7a7b89ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:02:27.059638 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:27.059613    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:02:27.350816 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:27.350791    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"]
Oct 30 20:02:27.353636 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:02:27.353606    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef621cd9_9b74_457a_a134_472f7a7b89ca.slice/crio-164fb3a976bf5cf75f8856bbb95c696bcc3cd9ed9cccf2e8510f409fc705892e WatchSource:0}: Error finding container 164fb3a976bf5cf75f8856bbb95c696bcc3cd9ed9cccf2e8510f409fc705892e: Status 404 returned error can't find the container with id 164fb3a976bf5cf75f8856bbb95c696bcc3cd9ed9cccf2e8510f409fc705892e
Oct 30 20:02:27.707838 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:27.707800    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"]
Oct 30 20:02:27.764724 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:27.764687    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz" event={"ID":"ef621cd9-9b74-457a-a134-472f7a7b89ca","Type":"ContainerStarted","Data":"164fb3a976bf5cf75f8856bbb95c696bcc3cd9ed9cccf2e8510f409fc705892e"}
Oct 30 20:02:28.774991 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:28.774956    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz" event={"ID":"ef621cd9-9b74-457a-a134-472f7a7b89ca","Type":"ContainerStarted","Data":"07fa8a6c13ff702f7ab83ab1bc3122590c69b8964ea24b9cc44fd79b16474143"}
Oct 30 20:02:28.775401 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:28.775024    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:02:28.802373 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:28.802326    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz" podStartSLOduration=2.377626292 podStartE2EDuration="2.802311794s" totalImagesPullingTime="424.685502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:28.798933817 +0000 UTC m=+1827.714540675" watchObservedRunningTime="2026-10-30 20:02:28.802311794 +0000 UTC m=+1827.717918650"
Oct 30 20:02:29.274672 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.274630    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr" podUID="5983703e-9048-445d-b9db-82e8bcff02a8" containerName="main" containerID="cri-o://535f188f0a815d753aa91937b97e45fa6149bd7d036328f696433741b576ae87" gracePeriod=45
Oct 30 20:02:29.568951 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.568925    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 20:02:29.714171 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.714123    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-model-cache\") pod \"5983703e-9048-445d-b9db-82e8bcff02a8\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") "
Oct 30 20:02:29.714345 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.714209    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-dshm\") pod \"5983703e-9048-445d-b9db-82e8bcff02a8\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") "
Oct 30 20:02:29.714345 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.714271    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5983703e-9048-445d-b9db-82e8bcff02a8-kube-api-access-6h7fs\" (UniqueName: \"kubernetes.io/projected/5983703e-9048-445d-b9db-82e8bcff02a8-kube-api-access-6h7fs\") pod \"5983703e-9048-445d-b9db-82e8bcff02a8\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") "
Oct 30 20:02:29.714345 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.714333    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-tmp-dir\") pod \"5983703e-9048-445d-b9db-82e8bcff02a8\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") "
Oct 30 20:02:29.714531 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.714357    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5983703e-9048-445d-b9db-82e8bcff02a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/5983703e-9048-445d-b9db-82e8bcff02a8-tls-certs\") pod \"5983703e-9048-445d-b9db-82e8bcff02a8\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") "
Oct 30 20:02:29.714531 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.714381    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-home\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-home\") pod \"5983703e-9048-445d-b9db-82e8bcff02a8\" (UID: \"5983703e-9048-445d-b9db-82e8bcff02a8\") "
Oct 30 20:02:29.714531 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.714444    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-model-cache" pod "5983703e-9048-445d-b9db-82e8bcff02a8" (UID: "5983703e-9048-445d-b9db-82e8bcff02a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:29.714717 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.714683    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-home" pod "5983703e-9048-445d-b9db-82e8bcff02a8" (UID: "5983703e-9048-445d-b9db-82e8bcff02a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:29.714717 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.714696    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-tmp-dir" pod "5983703e-9048-445d-b9db-82e8bcff02a8" (UID: "5983703e-9048-445d-b9db-82e8bcff02a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:29.715476 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.715435    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.715637 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.715621    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.715685 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.715643    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.716802 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.716775    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5983703e-9048-445d-b9db-82e8bcff02a8-tls-certs" pod "5983703e-9048-445d-b9db-82e8bcff02a8" (UID: "5983703e-9048-445d-b9db-82e8bcff02a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:29.716802 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.716789    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-dshm" pod "5983703e-9048-445d-b9db-82e8bcff02a8" (UID: "5983703e-9048-445d-b9db-82e8bcff02a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:29.716924 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.716838    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5983703e-9048-445d-b9db-82e8bcff02a8-kube-api-access-6h7fs" pod "5983703e-9048-445d-b9db-82e8bcff02a8" (UID: "5983703e-9048-445d-b9db-82e8bcff02a8"). InnerVolumeSpecName "kube-api-access-6h7fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:29.783480 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.783454    6030 generic.go:356] "Generic (PLEG): container finished" podID="5983703e-9048-445d-b9db-82e8bcff02a8" containerID="535f188f0a815d753aa91937b97e45fa6149bd7d036328f696433741b576ae87" exitCode=0
Oct 30 20:02:29.783880 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.783494    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"
Oct 30 20:02:29.783880 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.783533    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr" event={"ID":"5983703e-9048-445d-b9db-82e8bcff02a8","Type":"ContainerDied","Data":"535f188f0a815d753aa91937b97e45fa6149bd7d036328f696433741b576ae87"}
Oct 30 20:02:29.783880 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.783581    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr" event={"ID":"5983703e-9048-445d-b9db-82e8bcff02a8","Type":"ContainerDied","Data":"2c5347ade8098ff45136d3b0e2de09156c71914b195cad75ccf2b84094e5ef31"}
Oct 30 20:02:29.783880 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.783601    6030 scope.go:122] "RemoveContainer" containerID="535f188f0a815d753aa91937b97e45fa6149bd7d036328f696433741b576ae87"
Oct 30 20:02:29.794912 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.794895    6030 scope.go:122] "RemoveContainer" containerID="535f188f0a815d753aa91937b97e45fa6149bd7d036328f696433741b576ae87"
Oct 30 20:02:29.795221 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:02:29.795201    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535f188f0a815d753aa91937b97e45fa6149bd7d036328f696433741b576ae87\": container with ID starting with 535f188f0a815d753aa91937b97e45fa6149bd7d036328f696433741b576ae87 not found: ID does not exist" containerID="535f188f0a815d753aa91937b97e45fa6149bd7d036328f696433741b576ae87"
Oct 30 20:02:29.795289 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.795231    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535f188f0a815d753aa91937b97e45fa6149bd7d036328f696433741b576ae87"} err="failed to get container status \"535f188f0a815d753aa91937b97e45fa6149bd7d036328f696433741b576ae87\": rpc error: code = NotFound desc = could not find container \"535f188f0a815d753aa91937b97e45fa6149bd7d036328f696433741b576ae87\": container with ID starting with 535f188f0a815d753aa91937b97e45fa6149bd7d036328f696433741b576ae87 not found: ID does not exist"
Oct 30 20:02:29.811912 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.811884    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"]
Oct 30 20:02:29.816557 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.816534    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-7eaeed/auth-test-model-7eaeed-kserve-786b6f8cc7-k4xrr"]
Oct 30 20:02:29.817183 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.817166    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5983703e-9048-445d-b9db-82e8bcff02a8-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.817234 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.817187    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h7fs\" (UniqueName: \"kubernetes.io/projected/5983703e-9048-445d-b9db-82e8bcff02a8-kube-api-access-6h7fs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.817234 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:29.817198    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5983703e-9048-445d-b9db-82e8bcff02a8-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:31.203458 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:31.203423    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5983703e-9048-445d-b9db-82e8bcff02a8" path="/var/lib/kubelet/pods/5983703e-9048-445d-b9db-82e8bcff02a8/volumes"
Oct 30 20:02:33.367765 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.367718    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl" podUID="955ca10f-69cc-4cef-9c87-52beabd36dfa" containerName="main" containerID="cri-o://3faba949020f4462db0ece1bd9564b2d0dbf61891f4807651b42ddc387fa45d2" gracePeriod=45
Oct 30 20:02:33.640510 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.640477    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 20:02:33.678746 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.678710    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/955ca10f-69cc-4cef-9c87-52beabd36dfa-tls-certs\" (UniqueName: \"kubernetes.io/secret/955ca10f-69cc-4cef-9c87-52beabd36dfa-tls-certs\") pod \"955ca10f-69cc-4cef-9c87-52beabd36dfa\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") "
Oct 30 20:02:33.678906 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.678766    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-home\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-home\") pod \"955ca10f-69cc-4cef-9c87-52beabd36dfa\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") "
Oct 30 20:02:33.678906 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.678849    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-dshm\") pod \"955ca10f-69cc-4cef-9c87-52beabd36dfa\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") "
Oct 30 20:02:33.679039 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.678922    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-model-cache\") pod \"955ca10f-69cc-4cef-9c87-52beabd36dfa\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") "
Oct 30 20:02:33.679122 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.679070    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-tmp-dir\") pod \"955ca10f-69cc-4cef-9c87-52beabd36dfa\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") "
Oct 30 20:02:33.679122 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.679103    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955ca10f-69cc-4cef-9c87-52beabd36dfa-kube-api-access-qvvql\" (UniqueName: \"kubernetes.io/projected/955ca10f-69cc-4cef-9c87-52beabd36dfa-kube-api-access-qvvql\") pod \"955ca10f-69cc-4cef-9c87-52beabd36dfa\" (UID: \"955ca10f-69cc-4cef-9c87-52beabd36dfa\") "
Oct 30 20:02:33.679236 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.679120    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-home" pod "955ca10f-69cc-4cef-9c87-52beabd36dfa" (UID: "955ca10f-69cc-4cef-9c87-52beabd36dfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:33.679417 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.679338    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-tmp-dir" pod "955ca10f-69cc-4cef-9c87-52beabd36dfa" (UID: "955ca10f-69cc-4cef-9c87-52beabd36dfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:33.679689 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.679261    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-model-cache" pod "955ca10f-69cc-4cef-9c87-52beabd36dfa" (UID: "955ca10f-69cc-4cef-9c87-52beabd36dfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:33.679880 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.679858    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:33.680026 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.679883    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:33.680026 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.679897    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:33.681823 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.681798    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/955ca10f-69cc-4cef-9c87-52beabd36dfa-tls-certs" pod "955ca10f-69cc-4cef-9c87-52beabd36dfa" (UID: "955ca10f-69cc-4cef-9c87-52beabd36dfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:33.682376 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.682351    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955ca10f-69cc-4cef-9c87-52beabd36dfa-kube-api-access-qvvql" pod "955ca10f-69cc-4cef-9c87-52beabd36dfa" (UID: "955ca10f-69cc-4cef-9c87-52beabd36dfa"). InnerVolumeSpecName "kube-api-access-qvvql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:33.682542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.682524    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-dshm" pod "955ca10f-69cc-4cef-9c87-52beabd36dfa" (UID: "955ca10f-69cc-4cef-9c87-52beabd36dfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:33.774316 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.774281    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"]
Oct 30 20:02:33.775404 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.775383    6030 cpu_manager.go:414] "containerMap: removing container" podUID="955ca10f-69cc-4cef-9c87-52beabd36dfa" containerName="main"
Oct 30 20:02:33.775492 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.775406    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955ca10f-69cc-4cef-9c87-52beabd36dfa" containerName="main"
Oct 30 20:02:33.775492 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.775425    6030 cpu_manager.go:414] "containerMap: removing container" podUID="5983703e-9048-445d-b9db-82e8bcff02a8" containerName="main"
Oct 30 20:02:33.775492 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.775432    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5983703e-9048-445d-b9db-82e8bcff02a8" containerName="main"
Oct 30 20:02:33.775628 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.775615    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="955ca10f-69cc-4cef-9c87-52beabd36dfa" containerName="main"
Oct 30 20:02:33.775660 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.775635    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="5983703e-9048-445d-b9db-82e8bcff02a8" containerName="main"
Oct 30 20:02:33.779921 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.779901    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:02:33.780839 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.780812    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvvql\" (UniqueName: \"kubernetes.io/projected/955ca10f-69cc-4cef-9c87-52beabd36dfa-kube-api-access-qvvql\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:33.780925 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.780844    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955ca10f-69cc-4cef-9c87-52beabd36dfa-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:33.780925 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.780860    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955ca10f-69cc-4cef-9c87-52beabd36dfa-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:33.786457 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.784934    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-222480-serving-cert\""
Oct 30 20:02:33.792087 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.792061    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"]
Oct 30 20:02:33.817424 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.817381    6030 generic.go:356] "Generic (PLEG): container finished" podID="955ca10f-69cc-4cef-9c87-52beabd36dfa" containerID="3faba949020f4462db0ece1bd9564b2d0dbf61891f4807651b42ddc387fa45d2" exitCode=0
Oct 30 20:02:33.817558 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.817479    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl" event={"ID":"955ca10f-69cc-4cef-9c87-52beabd36dfa","Type":"ContainerDied","Data":"3faba949020f4462db0ece1bd9564b2d0dbf61891f4807651b42ddc387fa45d2"}
Oct 30 20:02:33.817558 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.817497    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"
Oct 30 20:02:33.817558 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.817515    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl" event={"ID":"955ca10f-69cc-4cef-9c87-52beabd36dfa","Type":"ContainerDied","Data":"d7efc7a21f885c6f80643177ee5a635d8c238194ce65c134387da0c0556674c7"}
Oct 30 20:02:33.817558 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.817539    6030 scope.go:122] "RemoveContainer" containerID="3faba949020f4462db0ece1bd9564b2d0dbf61891f4807651b42ddc387fa45d2"
Oct 30 20:02:33.828809 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.828695    6030 scope.go:122] "RemoveContainer" containerID="3faba949020f4462db0ece1bd9564b2d0dbf61891f4807651b42ddc387fa45d2"
Oct 30 20:02:33.829432 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:02:33.829411    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3faba949020f4462db0ece1bd9564b2d0dbf61891f4807651b42ddc387fa45d2\": container with ID starting with 3faba949020f4462db0ece1bd9564b2d0dbf61891f4807651b42ddc387fa45d2 not found: ID does not exist" containerID="3faba949020f4462db0ece1bd9564b2d0dbf61891f4807651b42ddc387fa45d2"
Oct 30 20:02:33.829566 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.829438    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3faba949020f4462db0ece1bd9564b2d0dbf61891f4807651b42ddc387fa45d2"} err="failed to get container status \"3faba949020f4462db0ece1bd9564b2d0dbf61891f4807651b42ddc387fa45d2\": rpc error: code = NotFound desc = could not find container \"3faba949020f4462db0ece1bd9564b2d0dbf61891f4807651b42ddc387fa45d2\": container with ID starting with 3faba949020f4462db0ece1bd9564b2d0dbf61891f4807651b42ddc387fa45d2 not found: ID does not exist"
Oct 30 20:02:33.855711 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.855677    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"]
Oct 30 20:02:33.865896 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.865874    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-981ee5/auth-test-model-981ee5-kserve-6655d955b9-5gcnl"]
Oct 30 20:02:33.881764 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.881740    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-maas-api-tls\") pod \"maas-api-e2e-extmod-a-222480-85c747d68c-wgv67\" (UID: \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:02:33.881880 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.881774    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stg9n\" (UniqueName: \"kubernetes.io/projected/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-kube-api-access-stg9n\") pod \"maas-api-e2e-extmod-a-222480-85c747d68c-wgv67\" (UID: \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:02:33.881880 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.881809    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-222480-85c747d68c-wgv67\" (UID: \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:02:33.882110 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.882090    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-metrics-certs\") pod \"maas-api-e2e-extmod-a-222480-85c747d68c-wgv67\" (UID: \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:02:33.986225 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.984114    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-metrics-certs\") pod \"maas-api-e2e-extmod-a-222480-85c747d68c-wgv67\" (UID: \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:02:33.986225 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.984279    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-maas-api-tls\") pod \"maas-api-e2e-extmod-a-222480-85c747d68c-wgv67\" (UID: \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:02:33.986225 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.984310    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stg9n\" (UniqueName: \"kubernetes.io/projected/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-kube-api-access-stg9n\") pod \"maas-api-e2e-extmod-a-222480-85c747d68c-wgv67\" (UID: \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:02:33.986225 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.984368    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-222480-85c747d68c-wgv67\" (UID: \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:02:33.986225 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:33.985716    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-222480-85c747d68c-wgv67\" (UID: \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:02:34.006154 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:34.006122    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-metrics-certs\") pod \"maas-api-e2e-extmod-a-222480-85c747d68c-wgv67\" (UID: \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:02:34.011031 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:34.010674    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stg9n\" (UniqueName: \"kubernetes.io/projected/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-kube-api-access-stg9n\") pod \"maas-api-e2e-extmod-a-222480-85c747d68c-wgv67\" (UID: \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:02:34.011679 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:34.011654    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-maas-api-tls\") pod \"maas-api-e2e-extmod-a-222480-85c747d68c-wgv67\" (UID: \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:02:34.108348 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:34.108300    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:02:34.304287 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:02:34.304242    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:02:34.454748 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:34.454717    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"]
Oct 30 20:02:34.789797 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:34.789719    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:02:34.834620 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:34.834578    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67" event={"ID":"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00","Type":"ContainerStarted","Data":"20ad82e4e6a36f3bd675bc03388b93e8948388a893c20e0bcf60dcc43391f9ed"}
Oct 30 20:02:35.101636 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:35.101561    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"]
Oct 30 20:02:35.101797 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:35.101775    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg" podUID="715973cd-8d23-4a57-b8ca-d22f81ac95a5" containerName="istio-proxy" containerID="cri-o://69c922a197baffb753db4989ec1e094ba25d04974711e1b197d67d25a277c6b9" gracePeriod=30
Oct 30 20:02:35.202877 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:35.202845    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955ca10f-69cc-4cef-9c87-52beabd36dfa" path="/var/lib/kubelet/pods/955ca10f-69cc-4cef-9c87-52beabd36dfa/volumes"
Oct 30 20:02:35.850166 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:35.850133    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67" event={"ID":"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00","Type":"ContainerStarted","Data":"854688ef58f98ff8e6ff13355fb37496124161d86eb296dd733f837aa7560232"}
Oct 30 20:02:35.850558 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:35.850267    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:02:35.883005 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:35.882958    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67" podStartSLOduration=2.4614800150000002 podStartE2EDuration="2.882939984s" totalImagesPullingTime="421.459969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:35.880889045 +0000 UTC m=+1834.796495901" watchObservedRunningTime="2026-10-30 20:02:35.882939984 +0000 UTC m=+1834.798546840"
Oct 30 20:02:36.757408 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:36.757370    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"
Oct 30 20:02:36.757555 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:36.757526    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"
Oct 30 20:02:39.788769 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:39.780689    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"]
Oct 30 20:02:39.788769 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:39.780918    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq" podUID="487f842d-ad13-45d8-8b55-fbf448ec511d" containerName="istio-proxy" containerID="cri-o://f33d47337250393337f478216b15cc0b25421f3e812abd6dcf960721deae5dde" gracePeriod=30
Oct 30 20:02:40.230325 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.230279    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"]
Oct 30 20:02:40.235843 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.235818    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.243075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.243002    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-68685b-openshift-default-dockercfg-2tqhz\""
Oct 30 20:02:40.248860 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.248506    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"]
Oct 30 20:02:40.359068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.339795    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-credential-socket\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.359068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.339862    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49k9x\" (UniqueName: \"kubernetes.io/projected/d4849f63-03b7-4090-bbc9-c0039ea48195-kube-api-access-49k9x\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.359068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.339902    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-podinfo\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.359068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.339981    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-data\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.359068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.340039    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-token\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.359068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.340123    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-workload-socket\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.359068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.340342    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-workload-certs\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.359068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.340387    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-envoy\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.359068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.344122    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4849f63-03b7-4090-bbc9-c0039ea48195-istiod-ca-cert\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.540331 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.490399    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-workload-certs\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.540331 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.490451    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-envoy\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.540331 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.490541    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4849f63-03b7-4090-bbc9-c0039ea48195-istiod-ca-cert\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.540331 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.490612    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-credential-socket\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.540331 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.490640    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49k9x\" (UniqueName: \"kubernetes.io/projected/d4849f63-03b7-4090-bbc9-c0039ea48195-kube-api-access-49k9x\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.540331 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.490696    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-podinfo\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.540331 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.490760    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-data\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.540331 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.490815    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-token\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.540331 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.490881    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-workload-socket\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.540331 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.511411    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-workload-socket\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.540331 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.511460    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-credential-socket\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.540331 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.511725    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-workload-certs\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.540331 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.535414    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-data\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.554336 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.552598    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4849f63-03b7-4090-bbc9-c0039ea48195-istiod-ca-cert\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.574843 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.574807    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-podinfo\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.592549 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.579031    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-envoy\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.592549 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.581181    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49k9x\" (UniqueName: \"kubernetes.io/projected/d4849f63-03b7-4090-bbc9-c0039ea48195-kube-api-access-49k9x\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.593971 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.593534    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-token\") pod \"e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") " pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.620684 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.620653    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 20:02:40.694577 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.694542    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-data\") pod \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") "
Oct 30 20:02:40.694956 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.694593    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istiod-ca-cert\") pod \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") "
Oct 30 20:02:40.694956 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.694636    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-token\" (UniqueName: \"kubernetes.io/projected/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-token\") pod \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") "
Oct 30 20:02:40.694956 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.694715    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715973cd-8d23-4a57-b8ca-d22f81ac95a5-kube-api-access-r2j88\" (UniqueName: \"kubernetes.io/projected/715973cd-8d23-4a57-b8ca-d22f81ac95a5-kube-api-access-r2j88\") pod \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") "
Oct 30 20:02:40.694956 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.694884    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-data" pod "715973cd-8d23-4a57-b8ca-d22f81ac95a5" (UID: "715973cd-8d23-4a57-b8ca-d22f81ac95a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:40.695253 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.695017    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-podinfo\") pod \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") "
Oct 30 20:02:40.695253 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.695184    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istiod-ca-cert" pod "715973cd-8d23-4a57-b8ca-d22f81ac95a5" (UID: "715973cd-8d23-4a57-b8ca-d22f81ac95a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:40.695397 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.695377    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-envoy\") pod \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") "
Oct 30 20:02:40.695560 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.695458    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-workload-certs\") pod \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") "
Oct 30 20:02:40.695560 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.695512    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-credential-socket\") pod \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") "
Oct 30 20:02:40.695708 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.695590    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-workload-socket\") pod \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\" (UID: \"715973cd-8d23-4a57-b8ca-d22f81ac95a5\") "
Oct 30 20:02:40.695793 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.695763    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-workload-certs" pod "715973cd-8d23-4a57-b8ca-d22f81ac95a5" (UID: "715973cd-8d23-4a57-b8ca-d22f81ac95a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:40.696253 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.696232    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:40.696340 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.696255    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:40.696340 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.696273    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:40.696340 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.696326    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-credential-socket" pod "715973cd-8d23-4a57-b8ca-d22f81ac95a5" (UID: "715973cd-8d23-4a57-b8ca-d22f81ac95a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:40.696445 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.696363    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-workload-socket" pod "715973cd-8d23-4a57-b8ca-d22f81ac95a5" (UID: "715973cd-8d23-4a57-b8ca-d22f81ac95a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:40.698184 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.698157    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715973cd-8d23-4a57-b8ca-d22f81ac95a5-kube-api-access-r2j88" pod "715973cd-8d23-4a57-b8ca-d22f81ac95a5" (UID: "715973cd-8d23-4a57-b8ca-d22f81ac95a5"). InnerVolumeSpecName "kube-api-access-r2j88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:40.698561 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.698531    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-token" pod "715973cd-8d23-4a57-b8ca-d22f81ac95a5" (UID: "715973cd-8d23-4a57-b8ca-d22f81ac95a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:40.698705 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.698652    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-podinfo" pod "715973cd-8d23-4a57-b8ca-d22f81ac95a5" (UID: "715973cd-8d23-4a57-b8ca-d22f81ac95a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:40.699989 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.699961    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-envoy" pod "715973cd-8d23-4a57-b8ca-d22f81ac95a5" (UID: "715973cd-8d23-4a57-b8ca-d22f81ac95a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:40.797511 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.797434    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:40.797511 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.797465    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2j88\" (UniqueName: \"kubernetes.io/projected/715973cd-8d23-4a57-b8ca-d22f81ac95a5-kube-api-access-r2j88\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:40.797511 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.797476    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:40.797511 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.797484    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:40.797511 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.797493    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:40.797511 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.797502    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715973cd-8d23-4a57-b8ca-d22f81ac95a5-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:40.864241 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.863341    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:40.906560 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.906534    6030 generic.go:356] "Generic (PLEG): container finished" podID="715973cd-8d23-4a57-b8ca-d22f81ac95a5" containerID="69c922a197baffb753db4989ec1e094ba25d04974711e1b197d67d25a277c6b9" exitCode=0
Oct 30 20:02:40.906682 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.906587    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg" event={"ID":"715973cd-8d23-4a57-b8ca-d22f81ac95a5","Type":"ContainerDied","Data":"69c922a197baffb753db4989ec1e094ba25d04974711e1b197d67d25a277c6b9"}
Oct 30 20:02:40.906682 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.906600    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"
Oct 30 20:02:40.906682 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.906638    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg" event={"ID":"715973cd-8d23-4a57-b8ca-d22f81ac95a5","Type":"ContainerDied","Data":"8cc5b0cf6928207bc695f667bf80c0ff9cfdce8a64d8b30d250ab5f2d99f3589"}
Oct 30 20:02:40.906682 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.906662    6030 scope.go:122] "RemoveContainer" containerID="69c922a197baffb753db4989ec1e094ba25d04974711e1b197d67d25a277c6b9"
Oct 30 20:02:40.921091 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.921029    6030 scope.go:122] "RemoveContainer" containerID="69c922a197baffb753db4989ec1e094ba25d04974711e1b197d67d25a277c6b9"
Oct 30 20:02:40.921552 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:02:40.921523    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c922a197baffb753db4989ec1e094ba25d04974711e1b197d67d25a277c6b9\": container with ID starting with 69c922a197baffb753db4989ec1e094ba25d04974711e1b197d67d25a277c6b9 not found: ID does not exist" containerID="69c922a197baffb753db4989ec1e094ba25d04974711e1b197d67d25a277c6b9"
Oct 30 20:02:40.921621 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.921564    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c922a197baffb753db4989ec1e094ba25d04974711e1b197d67d25a277c6b9"} err="failed to get container status \"69c922a197baffb753db4989ec1e094ba25d04974711e1b197d67d25a277c6b9\": rpc error: code = NotFound desc = could not find container \"69c922a197baffb753db4989ec1e094ba25d04974711e1b197d67d25a277c6b9\": container with ID starting with 69c922a197baffb753db4989ec1e094ba25d04974711e1b197d67d25a277c6b9 not found: ID does not exist"
Oct 30 20:02:40.965863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.965823    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"]
Oct 30 20:02:40.971658 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:40.969822    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-f6lcg"]
Oct 30 20:02:41.241160 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:41.241126    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="715973cd-8d23-4a57-b8ca-d22f81ac95a5" path="/var/lib/kubelet/pods/715973cd-8d23-4a57-b8ca-d22f81ac95a5/volumes"
Oct 30 20:02:41.270596 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:41.270567    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"]
Oct 30 20:02:41.271653 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:02:41.271629    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4849f63_03b7_4090_bbc9_c0039ea48195.slice/crio-893242d94606143eb285c1df331376be40423d08d4478d27d02b6dd52e3d377d WatchSource:0}: Error finding container 893242d94606143eb285c1df331376be40423d08d4478d27d02b6dd52e3d377d: Status 404 returned error can't find the container with id 893242d94606143eb285c1df331376be40423d08d4478d27d02b6dd52e3d377d
Oct 30 20:02:41.274917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:41.274880    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:02:41.275000 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:41.274949    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:02:41.275000 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:41.274979    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:02:41.869135 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:41.869103    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:02:41.919005 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:41.918965    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq" event={"ID":"d4849f63-03b7-4090-bbc9-c0039ea48195","Type":"ContainerStarted","Data":"ddbfcf9a0e65438f3149dc7368c25c4289329c9ddff73b0c7c61c76af5e76a73"}
Oct 30 20:02:41.919235 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:41.919030    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq" event={"ID":"d4849f63-03b7-4090-bbc9-c0039ea48195","Type":"ContainerStarted","Data":"893242d94606143eb285c1df331376be40423d08d4478d27d02b6dd52e3d377d"}
Oct 30 20:02:41.985854 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:41.985784    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq" podStartSLOduration=1.985765798 podStartE2EDuration="1.985765798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:41.979126313 +0000 UTC m=+1840.894733171" watchObservedRunningTime="2026-10-30 20:02:41.985765798 +0000 UTC m=+1840.901372655"
Oct 30 20:02:42.751269 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:42.750941    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6" podUID="0d51a759-22ec-4891-b58e-c8b72e85fbcc" containerName="main" containerID="cri-o://1e113ebf471f91d0a86b4bc92fa343205599070ece9cb2b6cf39f17a456c14ea" gracePeriod=45
Oct 30 20:02:42.865543 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:42.865509    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:42.872327 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:42.872301    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:42.928875 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:42.928840    6030 generic.go:356] "Generic (PLEG): container finished" podID="0d51a759-22ec-4891-b58e-c8b72e85fbcc" containerID="1e113ebf471f91d0a86b4bc92fa343205599070ece9cb2b6cf39f17a456c14ea" exitCode=0
Oct 30 20:02:42.929063 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:42.928916    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6" event={"ID":"0d51a759-22ec-4891-b58e-c8b72e85fbcc","Type":"ContainerDied","Data":"1e113ebf471f91d0a86b4bc92fa343205599070ece9cb2b6cf39f17a456c14ea"}
Oct 30 20:02:42.929301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:42.929278    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:42.930481 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:42.930460    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:02:43.099488 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.099460    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 20:02:43.134810 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.134731    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-tmp-dir\") pod \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") "
Oct 30 20:02:43.134987 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.134822    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-home\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-home\") pod \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") "
Oct 30 20:02:43.134987 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.134878    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d51a759-22ec-4891-b58e-c8b72e85fbcc-kube-api-access-lc79q\" (UniqueName: \"kubernetes.io/projected/0d51a759-22ec-4891-b58e-c8b72e85fbcc-kube-api-access-lc79q\") pod \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") "
Oct 30 20:02:43.134987 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.134920    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d51a759-22ec-4891-b58e-c8b72e85fbcc-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d51a759-22ec-4891-b58e-c8b72e85fbcc-tls-certs\") pod \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") "
Oct 30 20:02:43.134987 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.134982    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-dshm\") pod \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") "
Oct 30 20:02:43.135312 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.135293    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-model-cache\") pod \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\" (UID: \"0d51a759-22ec-4891-b58e-c8b72e85fbcc\") "
Oct 30 20:02:43.136175 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.136148    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-tmp-dir" pod "0d51a759-22ec-4891-b58e-c8b72e85fbcc" (UID: "0d51a759-22ec-4891-b58e-c8b72e85fbcc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:43.136423 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.136400    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-home" pod "0d51a759-22ec-4891-b58e-c8b72e85fbcc" (UID: "0d51a759-22ec-4891-b58e-c8b72e85fbcc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:43.136740 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.136714    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-model-cache" pod "0d51a759-22ec-4891-b58e-c8b72e85fbcc" (UID: "0d51a759-22ec-4891-b58e-c8b72e85fbcc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:43.139770 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.139739    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d51a759-22ec-4891-b58e-c8b72e85fbcc-tls-certs" pod "0d51a759-22ec-4891-b58e-c8b72e85fbcc" (UID: "0d51a759-22ec-4891-b58e-c8b72e85fbcc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:43.140459 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.140432    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d51a759-22ec-4891-b58e-c8b72e85fbcc-kube-api-access-lc79q" pod "0d51a759-22ec-4891-b58e-c8b72e85fbcc" (UID: "0d51a759-22ec-4891-b58e-c8b72e85fbcc"). InnerVolumeSpecName "kube-api-access-lc79q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:43.140795 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.140763    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-dshm" pod "0d51a759-22ec-4891-b58e-c8b72e85fbcc" (UID: "0d51a759-22ec-4891-b58e-c8b72e85fbcc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:43.237849 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.237815    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:43.237849 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.237851    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:43.238033 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.237867    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:43.238033 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.237884    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc79q\" (UniqueName: \"kubernetes.io/projected/0d51a759-22ec-4891-b58e-c8b72e85fbcc-kube-api-access-lc79q\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:43.238033 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.237900    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d51a759-22ec-4891-b58e-c8b72e85fbcc-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:43.238033 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.237914    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d51a759-22ec-4891-b58e-c8b72e85fbcc-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:43.937242 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.937206    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"
Oct 30 20:02:43.937725 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.937242    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6" event={"ID":"0d51a759-22ec-4891-b58e-c8b72e85fbcc","Type":"ContainerDied","Data":"61a60dfd5db25e17fbbd7615ed962a65b3b0ee335af8a0f95031980a310544d0"}
Oct 30 20:02:43.937725 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.937291    6030 scope.go:122] "RemoveContainer" containerID="1e113ebf471f91d0a86b4bc92fa343205599070ece9cb2b6cf39f17a456c14ea"
Oct 30 20:02:43.961436 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.961400    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"]
Oct 30 20:02:43.965231 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:43.965205    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b7431c/auth-test-model-b7431c-kserve-7b8fb979bf-b6wj6"]
Oct 30 20:02:44.947869 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:44.947837    6030 generic.go:356] "Generic (PLEG): container finished" podID="487f842d-ad13-45d8-8b55-fbf448ec511d" containerID="f33d47337250393337f478216b15cc0b25421f3e812abd6dcf960721deae5dde" exitCode=0
Oct 30 20:02:44.948293 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:44.947914    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq" event={"ID":"487f842d-ad13-45d8-8b55-fbf448ec511d","Type":"ContainerDied","Data":"f33d47337250393337f478216b15cc0b25421f3e812abd6dcf960721deae5dde"}
Oct 30 20:02:45.085212 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.085186    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 20:02:45.164528 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.164492    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-data\") pod \"487f842d-ad13-45d8-8b55-fbf448ec511d\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") "
Oct 30 20:02:45.164759 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.164547    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/487f842d-ad13-45d8-8b55-fbf448ec511d-kube-api-access-jgntr\" (UniqueName: \"kubernetes.io/projected/487f842d-ad13-45d8-8b55-fbf448ec511d-kube-api-access-jgntr\") pod \"487f842d-ad13-45d8-8b55-fbf448ec511d\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") "
Oct 30 20:02:45.164759 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.164591    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-workload-socket\") pod \"487f842d-ad13-45d8-8b55-fbf448ec511d\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") "
Oct 30 20:02:45.164759 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.164638    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-workload-certs\") pod \"487f842d-ad13-45d8-8b55-fbf448ec511d\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") "
Oct 30 20:02:45.164759 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.164674    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-token\" (UniqueName: \"kubernetes.io/projected/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-token\") pod \"487f842d-ad13-45d8-8b55-fbf448ec511d\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") "
Oct 30 20:02:45.164759 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.164720    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-envoy\") pod \"487f842d-ad13-45d8-8b55-fbf448ec511d\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") "
Oct 30 20:02:45.165111 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.164768    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-data" pod "487f842d-ad13-45d8-8b55-fbf448ec511d" (UID: "487f842d-ad13-45d8-8b55-fbf448ec511d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:45.165111 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.164798    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-credential-socket\") pod \"487f842d-ad13-45d8-8b55-fbf448ec511d\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") "
Oct 30 20:02:45.165111 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.164886    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-workload-socket" pod "487f842d-ad13-45d8-8b55-fbf448ec511d" (UID: "487f842d-ad13-45d8-8b55-fbf448ec511d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:45.165111 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.165094    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/487f842d-ad13-45d8-8b55-fbf448ec511d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487f842d-ad13-45d8-8b55-fbf448ec511d-istiod-ca-cert\") pod \"487f842d-ad13-45d8-8b55-fbf448ec511d\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") "
Oct 30 20:02:45.165318 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.165233    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-podinfo\") pod \"487f842d-ad13-45d8-8b55-fbf448ec511d\" (UID: \"487f842d-ad13-45d8-8b55-fbf448ec511d\") "
Oct 30 20:02:45.165318 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.165271    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-credential-socket" pod "487f842d-ad13-45d8-8b55-fbf448ec511d" (UID: "487f842d-ad13-45d8-8b55-fbf448ec511d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:45.165549 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.165520    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/487f842d-ad13-45d8-8b55-fbf448ec511d-istiod-ca-cert" pod "487f842d-ad13-45d8-8b55-fbf448ec511d" (UID: "487f842d-ad13-45d8-8b55-fbf448ec511d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:45.165687 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.165664    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-workload-certs" pod "487f842d-ad13-45d8-8b55-fbf448ec511d" (UID: "487f842d-ad13-45d8-8b55-fbf448ec511d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:45.165861 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.165840    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487f842d-ad13-45d8-8b55-fbf448ec511d-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:45.165861 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.165862    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:45.166017 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.165876    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:45.166017 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.165888    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:45.166017 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.165900    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:45.167290 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.167269    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487f842d-ad13-45d8-8b55-fbf448ec511d-kube-api-access-jgntr" pod "487f842d-ad13-45d8-8b55-fbf448ec511d" (UID: "487f842d-ad13-45d8-8b55-fbf448ec511d"). InnerVolumeSpecName "kube-api-access-jgntr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:45.167824 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.167793    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-envoy" pod "487f842d-ad13-45d8-8b55-fbf448ec511d" (UID: "487f842d-ad13-45d8-8b55-fbf448ec511d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:45.167997 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.167973    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-podinfo" pod "487f842d-ad13-45d8-8b55-fbf448ec511d" (UID: "487f842d-ad13-45d8-8b55-fbf448ec511d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:45.168235 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.168214    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-token" pod "487f842d-ad13-45d8-8b55-fbf448ec511d" (UID: "487f842d-ad13-45d8-8b55-fbf448ec511d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:45.202788 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.202754    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d51a759-22ec-4891-b58e-c8b72e85fbcc" path="/var/lib/kubelet/pods/0d51a759-22ec-4891-b58e-c8b72e85fbcc/volumes"
Oct 30 20:02:45.286156 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.276336    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:45.286156 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.276377    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:45.286156 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.276394    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgntr\" (UniqueName: \"kubernetes.io/projected/487f842d-ad13-45d8-8b55-fbf448ec511d-kube-api-access-jgntr\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:45.286156 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.276411    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487f842d-ad13-45d8-8b55-fbf448ec511d-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:45.590686 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.590644    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"]
Oct 30 20:02:45.591384 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.591364    6030 cpu_manager.go:414] "containerMap: removing container" podUID="715973cd-8d23-4a57-b8ca-d22f81ac95a5" containerName="istio-proxy"
Oct 30 20:02:45.591384 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.591384    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715973cd-8d23-4a57-b8ca-d22f81ac95a5" containerName="istio-proxy"
Oct 30 20:02:45.591658 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.591409    6030 cpu_manager.go:414] "containerMap: removing container" podUID="0d51a759-22ec-4891-b58e-c8b72e85fbcc" containerName="main"
Oct 30 20:02:45.591658 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.591416    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d51a759-22ec-4891-b58e-c8b72e85fbcc" containerName="main"
Oct 30 20:02:45.591658 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.591465    6030 cpu_manager.go:414] "containerMap: removing container" podUID="487f842d-ad13-45d8-8b55-fbf448ec511d" containerName="istio-proxy"
Oct 30 20:02:45.591658 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.591475    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="487f842d-ad13-45d8-8b55-fbf448ec511d" containerName="istio-proxy"
Oct 30 20:02:45.591658 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.591633    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="715973cd-8d23-4a57-b8ca-d22f81ac95a5" containerName="istio-proxy"
Oct 30 20:02:45.591658 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.591650    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d51a759-22ec-4891-b58e-c8b72e85fbcc" containerName="main"
Oct 30 20:02:45.591992 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.591670    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="487f842d-ad13-45d8-8b55-fbf448ec511d" containerName="istio-proxy"
Oct 30 20:02:45.596587 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.596566    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"
Oct 30 20:02:45.605864 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.605802    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-969878\""
Oct 30 20:02:45.607719 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.607699    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-969878-dockercfg-hwcfl\""
Oct 30 20:02:45.609142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.609121    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"]
Oct 30 20:02:45.636731 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.636699    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"]
Oct 30 20:02:45.640993 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.640973    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"
Oct 30 20:02:45.658909 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.658881    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"]
Oct 30 20:02:45.695309 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.682038    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb7mt\" (UniqueName: \"kubernetes.io/projected/0118f805-4deb-4de0-aaf2-cad5ba1ed2fa-kube-api-access-pb7mt\") pod \"payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm\" (UID: \"0118f805-4deb-4de0-aaf2-cad5ba1ed2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"
Oct 30 20:02:45.695309 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.682294    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0118f805-4deb-4de0-aaf2-cad5ba1ed2fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm\" (UID: \"0118f805-4deb-4de0-aaf2-cad5ba1ed2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"
Oct 30 20:02:45.794144 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.789523    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fbd9f0c-7c56-406e-8dd9-0770f49f4878-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb\" (UID: \"3fbd9f0c-7c56-406e-8dd9-0770f49f4878\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"
Oct 30 20:02:45.794144 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.789626    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb7mt\" (UniqueName: \"kubernetes.io/projected/0118f805-4deb-4de0-aaf2-cad5ba1ed2fa-kube-api-access-pb7mt\") pod \"payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm\" (UID: \"0118f805-4deb-4de0-aaf2-cad5ba1ed2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"
Oct 30 20:02:45.794144 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.789681    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjf8s\" (UniqueName: \"kubernetes.io/projected/3fbd9f0c-7c56-406e-8dd9-0770f49f4878-kube-api-access-xjf8s\") pod \"payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb\" (UID: \"3fbd9f0c-7c56-406e-8dd9-0770f49f4878\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"
Oct 30 20:02:45.794144 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.789831    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0118f805-4deb-4de0-aaf2-cad5ba1ed2fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm\" (UID: \"0118f805-4deb-4de0-aaf2-cad5ba1ed2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"
Oct 30 20:02:45.794144 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.790916    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0118f805-4deb-4de0-aaf2-cad5ba1ed2fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm\" (UID: \"0118f805-4deb-4de0-aaf2-cad5ba1ed2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"
Oct 30 20:02:45.815900 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.815856    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb7mt\" (UniqueName: \"kubernetes.io/projected/0118f805-4deb-4de0-aaf2-cad5ba1ed2fa-kube-api-access-pb7mt\") pod \"payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm\" (UID: \"0118f805-4deb-4de0-aaf2-cad5ba1ed2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"
Oct 30 20:02:45.891721 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.891683    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fbd9f0c-7c56-406e-8dd9-0770f49f4878-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb\" (UID: \"3fbd9f0c-7c56-406e-8dd9-0770f49f4878\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"
Oct 30 20:02:45.891922 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.891800    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjf8s\" (UniqueName: \"kubernetes.io/projected/3fbd9f0c-7c56-406e-8dd9-0770f49f4878-kube-api-access-xjf8s\") pod \"payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb\" (UID: \"3fbd9f0c-7c56-406e-8dd9-0770f49f4878\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"
Oct 30 20:02:45.892872 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.892843    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fbd9f0c-7c56-406e-8dd9-0770f49f4878-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb\" (UID: \"3fbd9f0c-7c56-406e-8dd9-0770f49f4878\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"
Oct 30 20:02:45.906163 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.906136    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjf8s\" (UniqueName: \"kubernetes.io/projected/3fbd9f0c-7c56-406e-8dd9-0770f49f4878-kube-api-access-xjf8s\") pod \"payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb\" (UID: \"3fbd9f0c-7c56-406e-8dd9-0770f49f4878\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"
Oct 30 20:02:45.912134 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.912112    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"
Oct 30 20:02:45.959944 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.959913    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"
Oct 30 20:02:45.960470 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.960250    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq" event={"ID":"487f842d-ad13-45d8-8b55-fbf448ec511d","Type":"ContainerDied","Data":"3d87131ddace5ab54b134fb40c8a49bf13320a4606ee5986481fb46180248170"}
Oct 30 20:02:45.960470 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.960322    6030 scope.go:122] "RemoveContainer" containerID="f33d47337250393337f478216b15cc0b25421f3e812abd6dcf960721deae5dde"
Oct 30 20:02:45.960470 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.960383    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"
Oct 30 20:02:45.989562 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.988259    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"]
Oct 30 20:02:45.999495 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:45.999455    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-7cmsq"]
Oct 30 20:02:46.207335 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.207307    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"]
Oct 30 20:02:46.207617 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:02:46.207599    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0118f805_4deb_4de0_aaf2_cad5ba1ed2fa.slice/crio-bbb6ab8b85645d9271f6eed81cd3cc63b8813a84c40e97a529b151e984ca3be1 WatchSource:0}: Error finding container bbb6ab8b85645d9271f6eed81cd3cc63b8813a84c40e97a529b151e984ca3be1: Status 404 returned error can't find the container with id bbb6ab8b85645d9271f6eed81cd3cc63b8813a84c40e97a529b151e984ca3be1
Oct 30 20:02:46.294668 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.294640    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"]
Oct 30 20:02:46.295689 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:02:46.295663    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbd9f0c_7c56_406e_8dd9_0770f49f4878.slice/crio-f539596b3906e0dba5908e1a3828d1a3f77adc8d1ea710829f6856b376bf5a11 WatchSource:0}: Error finding container f539596b3906e0dba5908e1a3828d1a3f77adc8d1ea710829f6856b376bf5a11: Status 404 returned error can't find the container with id f539596b3906e0dba5908e1a3828d1a3f77adc8d1ea710829f6856b376bf5a11
Oct 30 20:02:46.562100 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.562012    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"]
Oct 30 20:02:46.566367 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.566351    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"
Oct 30 20:02:46.574888 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.574302    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-68685b\""
Oct 30 20:02:46.575099 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.575062    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-68685b-dockercfg-xx4zg\""
Oct 30 20:02:46.578320 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.578300    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"]
Oct 30 20:02:46.586929 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.586901    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"]
Oct 30 20:02:46.592190 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.592170    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"
Oct 30 20:02:46.607922 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.607898    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"]
Oct 30 20:02:46.721072 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.714209    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4t8c\" (UniqueName: \"kubernetes.io/projected/64ec049e-096e-4146-8ad7-4d6a7398438f-kube-api-access-h4t8c\") pod \"payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27\" (UID: \"64ec049e-096e-4146-8ad7-4d6a7398438f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"
Oct 30 20:02:46.721072 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.714280    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ec049e-096e-4146-8ad7-4d6a7398438f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27\" (UID: \"64ec049e-096e-4146-8ad7-4d6a7398438f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"
Oct 30 20:02:46.721072 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.714349    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnstk\" (UniqueName: \"kubernetes.io/projected/b13ed493-c629-4370-9810-7dc11c243f7f-kube-api-access-vnstk\") pod \"payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj\" (UID: \"b13ed493-c629-4370-9810-7dc11c243f7f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"
Oct 30 20:02:46.721072 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.714483    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13ed493-c629-4370-9810-7dc11c243f7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj\" (UID: \"b13ed493-c629-4370-9810-7dc11c243f7f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"
Oct 30 20:02:46.819310 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.819228    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13ed493-c629-4370-9810-7dc11c243f7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj\" (UID: \"b13ed493-c629-4370-9810-7dc11c243f7f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"
Oct 30 20:02:46.819666 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.819641    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4t8c\" (UniqueName: \"kubernetes.io/projected/64ec049e-096e-4146-8ad7-4d6a7398438f-kube-api-access-h4t8c\") pod \"payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27\" (UID: \"64ec049e-096e-4146-8ad7-4d6a7398438f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"
Oct 30 20:02:46.819880 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.819861    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ec049e-096e-4146-8ad7-4d6a7398438f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27\" (UID: \"64ec049e-096e-4146-8ad7-4d6a7398438f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"
Oct 30 20:02:46.820081 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.820039    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnstk\" (UniqueName: \"kubernetes.io/projected/b13ed493-c629-4370-9810-7dc11c243f7f-kube-api-access-vnstk\") pod \"payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj\" (UID: \"b13ed493-c629-4370-9810-7dc11c243f7f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"
Oct 30 20:02:46.821016 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.820987    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13ed493-c629-4370-9810-7dc11c243f7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj\" (UID: \"b13ed493-c629-4370-9810-7dc11c243f7f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"
Oct 30 20:02:46.821570 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.821548    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ec049e-096e-4146-8ad7-4d6a7398438f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27\" (UID: \"64ec049e-096e-4146-8ad7-4d6a7398438f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"
Oct 30 20:02:46.834811 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.834574    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4t8c\" (UniqueName: \"kubernetes.io/projected/64ec049e-096e-4146-8ad7-4d6a7398438f-kube-api-access-h4t8c\") pod \"payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27\" (UID: \"64ec049e-096e-4146-8ad7-4d6a7398438f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"
Oct 30 20:02:46.835736 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.835717    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnstk\" (UniqueName: \"kubernetes.io/projected/b13ed493-c629-4370-9810-7dc11c243f7f-kube-api-access-vnstk\") pod \"payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj\" (UID: \"b13ed493-c629-4370-9810-7dc11c243f7f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"
Oct 30 20:02:46.882757 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.882729    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"
Oct 30 20:02:46.914103 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.914070    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"
Oct 30 20:02:46.976142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.975254    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb" event={"ID":"3fbd9f0c-7c56-406e-8dd9-0770f49f4878","Type":"ContainerStarted","Data":"6f6f089747e4170d17f004e1521b44faa435d2ea587ffe7cf8a0d883664f3caf"}
Oct 30 20:02:46.976142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.975305    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb" event={"ID":"3fbd9f0c-7c56-406e-8dd9-0770f49f4878","Type":"ContainerStarted","Data":"f539596b3906e0dba5908e1a3828d1a3f77adc8d1ea710829f6856b376bf5a11"}
Oct 30 20:02:46.976658 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.976601    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"
Oct 30 20:02:46.980507 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.980394    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm" event={"ID":"0118f805-4deb-4de0-aaf2-cad5ba1ed2fa","Type":"ContainerStarted","Data":"802b482b1cb150e52fc52450a4f7fbf82e86e9ddd6b3416c89e120f9e7401101"}
Oct 30 20:02:46.980507 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.980430    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm" event={"ID":"0118f805-4deb-4de0-aaf2-cad5ba1ed2fa","Type":"ContainerStarted","Data":"bbb6ab8b85645d9271f6eed81cd3cc63b8813a84c40e97a529b151e984ca3be1"}
Oct 30 20:02:46.981961 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:46.981883    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"
Oct 30 20:02:47.038851 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.037858    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb" podStartSLOduration=2.03783972 podStartE2EDuration="2.03783972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:47.009966952 +0000 UTC m=+1845.925573809" watchObservedRunningTime="2026-10-30 20:02:47.03783972 +0000 UTC m=+1845.953446577"
Oct 30 20:02:47.043763 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.042541    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm" podStartSLOduration=2.04253125 podStartE2EDuration="2.04253125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:47.035430249 +0000 UTC m=+1845.951037107" watchObservedRunningTime="2026-10-30 20:02:47.04253125 +0000 UTC m=+1845.958138107"
Oct 30 20:02:47.207009 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.206973    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="487f842d-ad13-45d8-8b55-fbf448ec511d" path="/var/lib/kubelet/pods/487f842d-ad13-45d8-8b55-fbf448ec511d/volumes"
Oct 30 20:02:47.222767 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.222735    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"]
Oct 30 20:02:47.228263 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:02:47.228206    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13ed493_c629_4370_9810_7dc11c243f7f.slice/crio-4f13e5977c39cd8b5c7d5a55dabf37f6c141077d4953185872b1d897ae5aedab WatchSource:0}: Error finding container 4f13e5977c39cd8b5c7d5a55dabf37f6c141077d4953185872b1d897ae5aedab: Status 404 returned error can't find the container with id 4f13e5977c39cd8b5c7d5a55dabf37f6c141077d4953185872b1d897ae5aedab
Oct 30 20:02:47.271268 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:02:47.271240    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ec049e_096e_4146_8ad7_4d6a7398438f.slice/crio-fb44ca453e790df0a2ea23306606b608babe29fb834e22eeef6577d956ed4367 WatchSource:0}: Error finding container fb44ca453e790df0a2ea23306606b608babe29fb834e22eeef6577d956ed4367: Status 404 returned error can't find the container with id fb44ca453e790df0a2ea23306606b608babe29fb834e22eeef6577d956ed4367
Oct 30 20:02:47.292312 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.292287    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"]
Oct 30 20:02:47.505615 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.505535    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"]
Oct 30 20:02:47.511226 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.511187    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:02:47.517787 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.517755    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-836d67-serving-cert\""
Oct 30 20:02:47.536956 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.536933    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"]
Oct 30 20:02:47.633749 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.633717    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/224a8d28-e274-485f-a31f-0b001e07a764-metrics-certs\") pod \"maas-api-e2e-autores-836d67-66f997457c-46vmp\" (UID: \"224a8d28-e274-485f-a31f-0b001e07a764\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:02:47.633888 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.633816    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224a8d28-e274-485f-a31f-0b001e07a764-maas-api-tls\") pod \"maas-api-e2e-autores-836d67-66f997457c-46vmp\" (UID: \"224a8d28-e274-485f-a31f-0b001e07a764\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:02:47.633888 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.633840    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224a8d28-e274-485f-a31f-0b001e07a764-openshift-service-ca\") pod \"maas-api-e2e-autores-836d67-66f997457c-46vmp\" (UID: \"224a8d28-e274-485f-a31f-0b001e07a764\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:02:47.634009 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.633989    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vvcj\" (UniqueName: \"kubernetes.io/projected/224a8d28-e274-485f-a31f-0b001e07a764-kube-api-access-9vvcj\") pod \"maas-api-e2e-autores-836d67-66f997457c-46vmp\" (UID: \"224a8d28-e274-485f-a31f-0b001e07a764\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:02:47.741069 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.736089    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224a8d28-e274-485f-a31f-0b001e07a764-maas-api-tls\") pod \"maas-api-e2e-autores-836d67-66f997457c-46vmp\" (UID: \"224a8d28-e274-485f-a31f-0b001e07a764\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:02:47.741069 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.736140    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224a8d28-e274-485f-a31f-0b001e07a764-openshift-service-ca\") pod \"maas-api-e2e-autores-836d67-66f997457c-46vmp\" (UID: \"224a8d28-e274-485f-a31f-0b001e07a764\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:02:47.741069 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.736289    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vvcj\" (UniqueName: \"kubernetes.io/projected/224a8d28-e274-485f-a31f-0b001e07a764-kube-api-access-9vvcj\") pod \"maas-api-e2e-autores-836d67-66f997457c-46vmp\" (UID: \"224a8d28-e274-485f-a31f-0b001e07a764\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:02:47.741069 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.736450    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/224a8d28-e274-485f-a31f-0b001e07a764-metrics-certs\") pod \"maas-api-e2e-autores-836d67-66f997457c-46vmp\" (UID: \"224a8d28-e274-485f-a31f-0b001e07a764\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:02:47.745016 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.744830    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224a8d28-e274-485f-a31f-0b001e07a764-openshift-service-ca\") pod \"maas-api-e2e-autores-836d67-66f997457c-46vmp\" (UID: \"224a8d28-e274-485f-a31f-0b001e07a764\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:02:47.751237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.751158    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/224a8d28-e274-485f-a31f-0b001e07a764-metrics-certs\") pod \"maas-api-e2e-autores-836d67-66f997457c-46vmp\" (UID: \"224a8d28-e274-485f-a31f-0b001e07a764\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:02:47.753233 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.753183    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224a8d28-e274-485f-a31f-0b001e07a764-maas-api-tls\") pod \"maas-api-e2e-autores-836d67-66f997457c-46vmp\" (UID: \"224a8d28-e274-485f-a31f-0b001e07a764\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:02:47.761693 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.761571    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vvcj\" (UniqueName: \"kubernetes.io/projected/224a8d28-e274-485f-a31f-0b001e07a764-kube-api-access-9vvcj\") pod \"maas-api-e2e-autores-836d67-66f997457c-46vmp\" (UID: \"224a8d28-e274-485f-a31f-0b001e07a764\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:02:47.831763 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.831729    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:02:47.994589 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.994553    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj" event={"ID":"b13ed493-c629-4370-9810-7dc11c243f7f","Type":"ContainerStarted","Data":"1526ae99c146c30697ae1ef3d53feb81fd9965c28353db880b941eb6dc9cd0c4"}
Oct 30 20:02:47.995070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.994614    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj" event={"ID":"b13ed493-c629-4370-9810-7dc11c243f7f","Type":"ContainerStarted","Data":"4f13e5977c39cd8b5c7d5a55dabf37f6c141077d4953185872b1d897ae5aedab"}
Oct 30 20:02:47.995070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.994700    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"
Oct 30 20:02:47.996886 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.996859    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27" event={"ID":"64ec049e-096e-4146-8ad7-4d6a7398438f","Type":"ContainerStarted","Data":"0d9b290fbc186d166112c3928171b2c115ae258d02279a36d310caa9d4b75e88"}
Oct 30 20:02:47.997016 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.996900    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27" event={"ID":"64ec049e-096e-4146-8ad7-4d6a7398438f","Type":"ContainerStarted","Data":"fb44ca453e790df0a2ea23306606b608babe29fb834e22eeef6577d956ed4367"}
Oct 30 20:02:47.997016 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:47.996995    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"
Oct 30 20:02:48.018526 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:48.018419    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj" podStartSLOduration=2.01840377 podStartE2EDuration="2.01840377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:48.015114464 +0000 UTC m=+1846.930721321" watchObservedRunningTime="2026-10-30 20:02:48.01840377 +0000 UTC m=+1846.934010627"
Oct 30 20:02:48.039689 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:48.039639    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27" podStartSLOduration=2.039626547 podStartE2EDuration="2.039626547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:48.037487636 +0000 UTC m=+1846.953094492" watchObservedRunningTime="2026-10-30 20:02:48.039626547 +0000 UTC m=+1846.955233408"
Oct 30 20:02:48.126340 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:48.126302    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"]
Oct 30 20:02:49.006614 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:49.006576    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp" event={"ID":"224a8d28-e274-485f-a31f-0b001e07a764","Type":"ContainerStarted","Data":"ff75bf561b520fdedc88903cf0cf86722f0d0d76108f055304181dd49d2c44a6"}
Oct 30 20:02:49.007161 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:49.006644    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp" event={"ID":"224a8d28-e274-485f-a31f-0b001e07a764","Type":"ContainerStarted","Data":"b9b307888cd35bc27d07a2cb1e0189608fd4ad231c9295044575c7ee3b986a76"}
Oct 30 20:02:49.034392 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:49.034333    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp" podStartSLOduration=1.581395326 podStartE2EDuration="2.034315809s" totalImagesPullingTime="452.920483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:49.028677225 +0000 UTC m=+1847.944284081" watchObservedRunningTime="2026-10-30 20:02:49.034315809 +0000 UTC m=+1847.949922666"
Oct 30 20:02:50.014570 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:50.014537    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:02:56.027496 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:56.027461    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:02:58.000464 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:58.000438    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"
Oct 30 20:02:59.010392 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:59.010359    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"
Oct 30 20:02:59.010853 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:59.010445    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"
Oct 30 20:02:59.010853 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:02:59.010530    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"
Oct 30 20:03:04.290369 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:03:04.290319    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:03:07.362959 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:07.362924    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"]
Oct 30 20:03:07.370830 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:07.370801    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:03:07.377239 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:07.377199    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-68685b-serving-cert\""
Oct 30 20:03:07.378740 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:07.378712    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"]
Oct 30 20:03:07.504908 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:07.504874    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f68b3878-c377-4994-9b24-1b2090038ba3-metrics-certs\") pod \"maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt\" (UID: \"f68b3878-c377-4994-9b24-1b2090038ba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:03:07.504908 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:07.504911    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f68b3878-c377-4994-9b24-1b2090038ba3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt\" (UID: \"f68b3878-c377-4994-9b24-1b2090038ba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:03:07.505198 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:07.505169    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68b3878-c377-4994-9b24-1b2090038ba3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt\" (UID: \"f68b3878-c377-4994-9b24-1b2090038ba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:03:07.505304 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:07.505288    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzxn2\" (UniqueName: \"kubernetes.io/projected/f68b3878-c377-4994-9b24-1b2090038ba3-kube-api-access-jzxn2\") pod \"maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt\" (UID: \"f68b3878-c377-4994-9b24-1b2090038ba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:03:07.606059 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:07.606014    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f68b3878-c377-4994-9b24-1b2090038ba3-metrics-certs\") pod \"maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt\" (UID: \"f68b3878-c377-4994-9b24-1b2090038ba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:03:07.606225 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:07.606081    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f68b3878-c377-4994-9b24-1b2090038ba3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt\" (UID: \"f68b3878-c377-4994-9b24-1b2090038ba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:03:07.606269 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:07.606236    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68b3878-c377-4994-9b24-1b2090038ba3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt\" (UID: \"f68b3878-c377-4994-9b24-1b2090038ba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:03:07.606336 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:07.606322    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzxn2\" (UniqueName: \"kubernetes.io/projected/f68b3878-c377-4994-9b24-1b2090038ba3-kube-api-access-jzxn2\") pod \"maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt\" (UID: \"f68b3878-c377-4994-9b24-1b2090038ba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:03:07.606941 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:07.606917    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68b3878-c377-4994-9b24-1b2090038ba3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt\" (UID: \"f68b3878-c377-4994-9b24-1b2090038ba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:03:07.608861 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:07.608837    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f68b3878-c377-4994-9b24-1b2090038ba3-metrics-certs\") pod \"maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt\" (UID: \"f68b3878-c377-4994-9b24-1b2090038ba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:03:07.608953 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:07.608837    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f68b3878-c377-4994-9b24-1b2090038ba3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt\" (UID: \"f68b3878-c377-4994-9b24-1b2090038ba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:03:07.623570 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:07.623547    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzxn2\" (UniqueName: \"kubernetes.io/projected/f68b3878-c377-4994-9b24-1b2090038ba3-kube-api-access-jzxn2\") pod \"maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt\" (UID: \"f68b3878-c377-4994-9b24-1b2090038ba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:03:07.702347 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:07.702307    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:03:08.179667 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:08.179622    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"]
Oct 30 20:03:09.213759 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:09.213732    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:03:09.214187 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:09.213780    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt" event={"ID":"f68b3878-c377-4994-9b24-1b2090038ba3","Type":"ContainerStarted","Data":"6f3ce1197f17ad5bbeeabe272607f3036d3e715117d1064c8d549b082e648de9"}
Oct 30 20:03:09.214187 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:09.213800    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt" event={"ID":"f68b3878-c377-4994-9b24-1b2090038ba3","Type":"ContainerStarted","Data":"89e44d042dda6a8ed5da399149aa8de71991cfedb1ab6cef41d3109610ec68dd"}
Oct 30 20:03:09.235076 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:09.235011    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt" podStartSLOduration=1.741183519 podStartE2EDuration="2.234997175s" totalImagesPullingTime="493.813656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:09.234327353 +0000 UTC m=+1868.149934210" watchObservedRunningTime="2026-10-30 20:03:09.234997175 +0000 UTC m=+1868.150604031"
Oct 30 20:03:15.235895 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:15.235810    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:03:19.346372 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:19.346341    6030 generic.go:356] "Generic (PLEG): container finished" podID="5075d107-d1a3-4f56-afd0-6be45cbda01d" containerID="28d5fb61c8d48c4c29739e388eee1b80641ecc70845eea1e282c0ff04af5b158" exitCode=0
Oct 30 20:03:19.346841 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:19.346378    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8" event={"ID":"5075d107-d1a3-4f56-afd0-6be45cbda01d","Type":"ContainerDied","Data":"28d5fb61c8d48c4c29739e388eee1b80641ecc70845eea1e282c0ff04af5b158"}
Oct 30 20:03:19.346841 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:19.346801    6030 scope.go:122] "RemoveContainer" containerID="28d5fb61c8d48c4c29739e388eee1b80641ecc70845eea1e282c0ff04af5b158"
Oct 30 20:03:19.348596 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:19.348562    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:03:19.348685 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:19.348635    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:03:19.348685 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:19.348666    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:03:20.355268 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:20.355229    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8" event={"ID":"5075d107-d1a3-4f56-afd0-6be45cbda01d","Type":"ContainerStarted","Data":"53b74d55c2fdeb0e080be3f797e078c087588b606c6b0a86069d67a497197f4a"}
Oct 30 20:03:20.438210 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:20.438180    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:03:20.438381 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:20.438347    6030 patch_prober.go:29] interesting pod/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 20:03:20.438426 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:20.438398    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8" podUID="5075d107-d1a3-4f56-afd0-6be45cbda01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 20:03:21.437697 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:21.437658    6030 patch_prober.go:29] interesting pod/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 20:03:21.438140 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:21.437727    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8" podUID="5075d107-d1a3-4f56-afd0-6be45cbda01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 20:03:22.437393 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:22.437357    6030 patch_prober.go:29] interesting pod/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 20:03:22.437608 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:22.437430    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8" podUID="5075d107-d1a3-4f56-afd0-6be45cbda01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 20:03:23.438354 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:23.438321    6030 patch_prober.go:29] interesting pod/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 20:03:23.438771 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:23.438391    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8" podUID="5075d107-d1a3-4f56-afd0-6be45cbda01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 20:03:24.438299 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:24.438265    6030 patch_prober.go:29] interesting pod/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 20:03:24.438475 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:24.438327    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8" podUID="5075d107-d1a3-4f56-afd0-6be45cbda01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 20:03:25.437562 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:25.437530    6030 patch_prober.go:29] interesting pod/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 20:03:25.437771 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:25.437582    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8" podUID="5075d107-d1a3-4f56-afd0-6be45cbda01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 20:03:25.524870 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:25.524834    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:03:26.442816 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:26.442777    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:03:26.443857 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:26.443832    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:03:34.318692 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:03:34.318654    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:03:42.552525 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:42.552486    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"]
Oct 30 20:03:42.575086 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:42.573220    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"]
Oct 30 20:03:42.670829 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:42.670790    6030 generic.go:356] "Generic (PLEG): container finished" podID="d4849f63-03b7-4090-bbc9-c0039ea48195" containerID="ddbfcf9a0e65438f3149dc7368c25c4289329c9ddff73b0c7c61c76af5e76a73" exitCode=0
Oct 30 20:03:42.671001 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:42.670880    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq" event={"ID":"d4849f63-03b7-4090-bbc9-c0039ea48195","Type":"ContainerDied","Data":"ddbfcf9a0e65438f3149dc7368c25c4289329c9ddff73b0c7c61c76af5e76a73"}
Oct 30 20:03:42.671459 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:42.671438    6030 scope.go:122] "RemoveContainer" containerID="ddbfcf9a0e65438f3149dc7368c25c4289329c9ddff73b0c7c61c76af5e76a73"
Oct 30 20:03:42.673793 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:42.673754    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:03:42.673889 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:42.673846    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:03:42.673970 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:42.673896    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:03:42.938073 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:42.934525    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:03:43.679519 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:43.679481    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq" event={"ID":"d4849f63-03b7-4090-bbc9-c0039ea48195","Type":"ContainerStarted","Data":"9600c4536bf04f200156d0c8828e879eaac5d01f6790290cf80fce931e410a35"}
Oct 30 20:03:43.865426 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:43.865394    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:03:43.867149 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:43.867116    6030 patch_prober.go:29] interesting pod/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 20:03:43.867300 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:43.867167    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq" podUID="d4849f63-03b7-4090-bbc9-c0039ea48195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 20:03:44.866330 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:44.866295    6030 patch_prober.go:29] interesting pod/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 20:03:44.866835 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:44.866358    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq" podUID="d4849f63-03b7-4090-bbc9-c0039ea48195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 20:03:44.868888 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:44.868860    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"]
Oct 30 20:03:44.896075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:44.895093    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"
Oct 30 20:03:44.907287 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:44.907257    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-2a2286\""
Oct 30 20:03:44.907550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:44.907531    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-2a2286-dockercfg-c9t7r\""
Oct 30 20:03:44.908433 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:44.908354    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"]
Oct 30 20:03:45.028070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:45.027815    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f76d83-cf47-4e1e-a66b-a6951bbc79cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7\" (UID: \"f5f76d83-cf47-4e1e-a66b-a6951bbc79cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"
Oct 30 20:03:45.028070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:45.027933    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qttc\" (UniqueName: \"kubernetes.io/projected/f5f76d83-cf47-4e1e-a66b-a6951bbc79cf-kube-api-access-8qttc\") pod \"payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7\" (UID: \"f5f76d83-cf47-4e1e-a66b-a6951bbc79cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"
Oct 30 20:03:45.135120 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:45.131919    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f76d83-cf47-4e1e-a66b-a6951bbc79cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7\" (UID: \"f5f76d83-cf47-4e1e-a66b-a6951bbc79cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"
Oct 30 20:03:45.135120 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:45.132037    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qttc\" (UniqueName: \"kubernetes.io/projected/f5f76d83-cf47-4e1e-a66b-a6951bbc79cf-kube-api-access-8qttc\") pod \"payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7\" (UID: \"f5f76d83-cf47-4e1e-a66b-a6951bbc79cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"
Oct 30 20:03:45.135120 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:45.133277    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f76d83-cf47-4e1e-a66b-a6951bbc79cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7\" (UID: \"f5f76d83-cf47-4e1e-a66b-a6951bbc79cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"
Oct 30 20:03:45.177859 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:45.177825    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qttc\" (UniqueName: \"kubernetes.io/projected/f5f76d83-cf47-4e1e-a66b-a6951bbc79cf-kube-api-access-8qttc\") pod \"payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7\" (UID: \"f5f76d83-cf47-4e1e-a66b-a6951bbc79cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"
Oct 30 20:03:45.224196 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:45.224142    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"
Oct 30 20:03:45.600219 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:45.598196    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"]
Oct 30 20:03:45.615944 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:03:45.615905    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f76d83_cf47_4e1e_a66b_a6951bbc79cf.slice/crio-289b5beee9ab62b0e967fdfeec8260908b1a81a82a6accd5ea8fef7663fc582a WatchSource:0}: Error finding container 289b5beee9ab62b0e967fdfeec8260908b1a81a82a6accd5ea8fef7663fc582a: Status 404 returned error can't find the container with id 289b5beee9ab62b0e967fdfeec8260908b1a81a82a6accd5ea8fef7663fc582a
Oct 30 20:03:45.730116 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:45.729561    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7" event={"ID":"f5f76d83-cf47-4e1e-a66b-a6951bbc79cf","Type":"ContainerStarted","Data":"289b5beee9ab62b0e967fdfeec8260908b1a81a82a6accd5ea8fef7663fc582a"}
Oct 30 20:03:45.866265 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:45.866182    6030 patch_prober.go:29] interesting pod/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 20:03:45.866265 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:45.866234    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq" podUID="d4849f63-03b7-4090-bbc9-c0039ea48195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 20:03:46.738414 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:46.738378    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7" event={"ID":"f5f76d83-cf47-4e1e-a66b-a6951bbc79cf","Type":"ContainerStarted","Data":"0da95a002d335d790eaa42a4d1c4cd2d3c9df23084665459b255e86a8b56594e"}
Oct 30 20:03:46.738591 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:46.738448    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"
Oct 30 20:03:46.767555 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:46.767509    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7" podStartSLOduration=2.7674948600000002 podStartE2EDuration="2.76749486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:46.763427608 +0000 UTC m=+1905.679034477" watchObservedRunningTime="2026-10-30 20:03:46.76749486 +0000 UTC m=+1905.683101716"
Oct 30 20:03:46.866340 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:46.866306    6030 patch_prober.go:29] interesting pod/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 20:03:46.866517 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:46.866359    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq" podUID="d4849f63-03b7-4090-bbc9-c0039ea48195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 20:03:47.553764 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:47.553714    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx" podUID="73882a2e-adf8-4aaa-9da6-49d5e8917248" containerName="payload-processing" containerID="cri-o://ba6c94bef6f2ab4f5c3d43645a0f437429d6c2f355811a609d776630e41de320" gracePeriod=25
Oct 30 20:03:47.581725 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:47.573912    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8" podUID="e1682349-9425-4aba-a4b6-01bbe7427c61" containerName="payload-pre-processing" containerID="cri-o://06f4bcfb01a968c81ddf60caf25a3c68e8c25600173a87970b23c1b13c3c1936" gracePeriod=25
Oct 30 20:03:47.748177 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:47.748136    6030 generic.go:356] "Generic (PLEG): container finished" podID="73882a2e-adf8-4aaa-9da6-49d5e8917248" containerID="ba6c94bef6f2ab4f5c3d43645a0f437429d6c2f355811a609d776630e41de320" exitCode=0
Oct 30 20:03:47.748328 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:47.748298    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx" event={"ID":"73882a2e-adf8-4aaa-9da6-49d5e8917248","Type":"ContainerDied","Data":"ba6c94bef6f2ab4f5c3d43645a0f437429d6c2f355811a609d776630e41de320"}
Oct 30 20:03:47.750378 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:47.750353    6030 generic.go:356] "Generic (PLEG): container finished" podID="e1682349-9425-4aba-a4b6-01bbe7427c61" containerID="06f4bcfb01a968c81ddf60caf25a3c68e8c25600173a87970b23c1b13c3c1936" exitCode=0
Oct 30 20:03:47.750506 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:47.750427    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8" event={"ID":"e1682349-9425-4aba-a4b6-01bbe7427c61","Type":"ContainerDied","Data":"06f4bcfb01a968c81ddf60caf25a3c68e8c25600173a87970b23c1b13c3c1936"}
Oct 30 20:03:47.866168 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:47.866132    6030 patch_prober.go:29] interesting pod/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 20:03:47.866334 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:47.866185    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq" podUID="d4849f63-03b7-4090-bbc9-c0039ea48195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 20:03:47.963897 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:47.962939    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"
Oct 30 20:03:48.010834 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.010779    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"
Oct 30 20:03:48.040306 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.040273    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73882a2e-adf8-4aaa-9da6-49d5e8917248-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73882a2e-adf8-4aaa-9da6-49d5e8917248-plugins-config-volume\") pod \"73882a2e-adf8-4aaa-9da6-49d5e8917248\" (UID: \"73882a2e-adf8-4aaa-9da6-49d5e8917248\") "
Oct 30 20:03:48.040610 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.040588    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73882a2e-adf8-4aaa-9da6-49d5e8917248-kube-api-access-7zptk\" (UniqueName: \"kubernetes.io/projected/73882a2e-adf8-4aaa-9da6-49d5e8917248-kube-api-access-7zptk\") pod \"73882a2e-adf8-4aaa-9da6-49d5e8917248\" (UID: \"73882a2e-adf8-4aaa-9da6-49d5e8917248\") "
Oct 30 20:03:48.041373 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.041349    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73882a2e-adf8-4aaa-9da6-49d5e8917248-plugins-config-volume" pod "73882a2e-adf8-4aaa-9da6-49d5e8917248" (UID: "73882a2e-adf8-4aaa-9da6-49d5e8917248"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:48.043968 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.043944    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73882a2e-adf8-4aaa-9da6-49d5e8917248-kube-api-access-7zptk" pod "73882a2e-adf8-4aaa-9da6-49d5e8917248" (UID: "73882a2e-adf8-4aaa-9da6-49d5e8917248"). InnerVolumeSpecName "kube-api-access-7zptk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:48.147964 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.147924    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1682349-9425-4aba-a4b6-01bbe7427c61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1682349-9425-4aba-a4b6-01bbe7427c61-plugins-config-volume\") pod \"e1682349-9425-4aba-a4b6-01bbe7427c61\" (UID: \"e1682349-9425-4aba-a4b6-01bbe7427c61\") "
Oct 30 20:03:48.148199 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.148178    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1682349-9425-4aba-a4b6-01bbe7427c61-kube-api-access-pb7f9\" (UniqueName: \"kubernetes.io/projected/e1682349-9425-4aba-a4b6-01bbe7427c61-kube-api-access-pb7f9\") pod \"e1682349-9425-4aba-a4b6-01bbe7427c61\" (UID: \"e1682349-9425-4aba-a4b6-01bbe7427c61\") "
Oct 30 20:03:48.148860 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.148772    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zptk\" (UniqueName: \"kubernetes.io/projected/73882a2e-adf8-4aaa-9da6-49d5e8917248-kube-api-access-7zptk\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:48.148860 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.148800    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73882a2e-adf8-4aaa-9da6-49d5e8917248-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:48.179090 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.168143    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1682349-9425-4aba-a4b6-01bbe7427c61-plugins-config-volume" pod "e1682349-9425-4aba-a4b6-01bbe7427c61" (UID: "e1682349-9425-4aba-a4b6-01bbe7427c61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:48.220240 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.220199    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1682349-9425-4aba-a4b6-01bbe7427c61-kube-api-access-pb7f9" pod "e1682349-9425-4aba-a4b6-01bbe7427c61" (UID: "e1682349-9425-4aba-a4b6-01bbe7427c61"). InnerVolumeSpecName "kube-api-access-pb7f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:48.250655 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.250624    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1682349-9425-4aba-a4b6-01bbe7427c61-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:48.250655 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.250652    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb7f9\" (UniqueName: \"kubernetes.io/projected/e1682349-9425-4aba-a4b6-01bbe7427c61-kube-api-access-pb7f9\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:48.758926 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.758893    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"
Oct 30 20:03:48.759139 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.758935    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8" event={"ID":"e1682349-9425-4aba-a4b6-01bbe7427c61","Type":"ContainerDied","Data":"ed642b5de27721562b2a88ad021deef415e5460ceca9b38299352e4371dfc887"}
Oct 30 20:03:48.759139 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.759002    6030 scope.go:122] "RemoveContainer" containerID="06f4bcfb01a968c81ddf60caf25a3c68e8c25600173a87970b23c1b13c3c1936"
Oct 30 20:03:48.760693 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.760675    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"
Oct 30 20:03:48.760799 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.760668    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx" event={"ID":"73882a2e-adf8-4aaa-9da6-49d5e8917248","Type":"ContainerDied","Data":"afdd17f4e0dc56a2927e99a3d0fb9bc71ee95041a563c1b311cc3f657f1c008f"}
Oct 30 20:03:48.797120 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.796344    6030 scope.go:122] "RemoveContainer" containerID="ba6c94bef6f2ab4f5c3d43645a0f437429d6c2f355811a609d776630e41de320"
Oct 30 20:03:48.819237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.819207    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"]
Oct 30 20:03:48.824438 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.824381    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-836d67-665d67575f-2mpk8"]
Oct 30 20:03:48.843781 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.843750    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"]
Oct 30 20:03:48.846733 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.846707    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-836d67-5d8bbbf5b8-x2dzx"]
Oct 30 20:03:48.866556 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.866517    6030 patch_prober.go:29] interesting pod/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 20:03:48.866733 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:48.866575    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq" podUID="d4849f63-03b7-4090-bbc9-c0039ea48195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 20:03:49.226188 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:49.226157    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73882a2e-adf8-4aaa-9da6-49d5e8917248" path="/var/lib/kubelet/pods/73882a2e-adf8-4aaa-9da6-49d5e8917248/volumes"
Oct 30 20:03:49.226870 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:49.226847    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1682349-9425-4aba-a4b6-01bbe7427c61" path="/var/lib/kubelet/pods/e1682349-9425-4aba-a4b6-01bbe7427c61/volumes"
Oct 30 20:03:49.872922 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:49.872893    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:03:49.874246 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:49.874223    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:03:49.874899 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:49.874879    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:03:57.753602 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:03:57.753573    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"
Oct 30 20:04:03.811911 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:03.811871    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"]
Oct 30 20:04:03.812575 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:03.812247    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp" podUID="224a8d28-e274-485f-a31f-0b001e07a764" containerName="maas-api" containerID="cri-o://ff75bf561b520fdedc88903cf0cf86722f0d0d76108f055304181dd49d2c44a6" gracePeriod=30
Oct 30 20:04:04.236187 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.234321    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:04:04.307069 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:04:04.307009    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:04:04.323846 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.323812    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/224a8d28-e274-485f-a31f-0b001e07a764-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224a8d28-e274-485f-a31f-0b001e07a764-maas-api-tls\") pod \"224a8d28-e274-485f-a31f-0b001e07a764\" (UID: \"224a8d28-e274-485f-a31f-0b001e07a764\") "
Oct 30 20:04:04.324020 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.323892    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/224a8d28-e274-485f-a31f-0b001e07a764-kube-api-access-9vvcj\" (UniqueName: \"kubernetes.io/projected/224a8d28-e274-485f-a31f-0b001e07a764-kube-api-access-9vvcj\") pod \"224a8d28-e274-485f-a31f-0b001e07a764\" (UID: \"224a8d28-e274-485f-a31f-0b001e07a764\") "
Oct 30 20:04:04.324020 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.323973    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/224a8d28-e274-485f-a31f-0b001e07a764-metrics-certs\" (UniqueName: \"kubernetes.io/secret/224a8d28-e274-485f-a31f-0b001e07a764-metrics-certs\") pod \"224a8d28-e274-485f-a31f-0b001e07a764\" (UID: \"224a8d28-e274-485f-a31f-0b001e07a764\") "
Oct 30 20:04:04.324181 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.324133    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/224a8d28-e274-485f-a31f-0b001e07a764-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224a8d28-e274-485f-a31f-0b001e07a764-openshift-service-ca\") pod \"224a8d28-e274-485f-a31f-0b001e07a764\" (UID: \"224a8d28-e274-485f-a31f-0b001e07a764\") "
Oct 30 20:04:04.325686 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.325655    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/224a8d28-e274-485f-a31f-0b001e07a764-openshift-service-ca" pod "224a8d28-e274-485f-a31f-0b001e07a764" (UID: "224a8d28-e274-485f-a31f-0b001e07a764"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:04.327888 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.327829    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/224a8d28-e274-485f-a31f-0b001e07a764-metrics-certs" pod "224a8d28-e274-485f-a31f-0b001e07a764" (UID: "224a8d28-e274-485f-a31f-0b001e07a764"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:04.328378 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.328345    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224a8d28-e274-485f-a31f-0b001e07a764-kube-api-access-9vvcj" pod "224a8d28-e274-485f-a31f-0b001e07a764" (UID: "224a8d28-e274-485f-a31f-0b001e07a764"). InnerVolumeSpecName "kube-api-access-9vvcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:04.330736 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.330713    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/224a8d28-e274-485f-a31f-0b001e07a764-maas-api-tls" pod "224a8d28-e274-485f-a31f-0b001e07a764" (UID: "224a8d28-e274-485f-a31f-0b001e07a764"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:04.430347 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.430228    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/224a8d28-e274-485f-a31f-0b001e07a764-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:04.430347 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.430271    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224a8d28-e274-485f-a31f-0b001e07a764-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:04.430347 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.430289    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224a8d28-e274-485f-a31f-0b001e07a764-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:04.430347 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.430305    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vvcj\" (UniqueName: \"kubernetes.io/projected/224a8d28-e274-485f-a31f-0b001e07a764-kube-api-access-9vvcj\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:04.879781 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.879745    6030 generic.go:356] "Generic (PLEG): container finished" podID="224a8d28-e274-485f-a31f-0b001e07a764" containerID="ff75bf561b520fdedc88903cf0cf86722f0d0d76108f055304181dd49d2c44a6" exitCode=0
Oct 30 20:04:04.880213 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.879806    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"
Oct 30 20:04:04.880213 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.879823    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp" event={"ID":"224a8d28-e274-485f-a31f-0b001e07a764","Type":"ContainerDied","Data":"ff75bf561b520fdedc88903cf0cf86722f0d0d76108f055304181dd49d2c44a6"}
Oct 30 20:04:04.880213 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.879913    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp" event={"ID":"224a8d28-e274-485f-a31f-0b001e07a764","Type":"ContainerDied","Data":"b9b307888cd35bc27d07a2cb1e0189608fd4ad231c9295044575c7ee3b986a76"}
Oct 30 20:04:04.880213 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.879944    6030 scope.go:122] "RemoveContainer" containerID="ff75bf561b520fdedc88903cf0cf86722f0d0d76108f055304181dd49d2c44a6"
Oct 30 20:04:04.891325 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.891307    6030 scope.go:122] "RemoveContainer" containerID="ff75bf561b520fdedc88903cf0cf86722f0d0d76108f055304181dd49d2c44a6"
Oct 30 20:04:04.891596 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:04:04.891576    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff75bf561b520fdedc88903cf0cf86722f0d0d76108f055304181dd49d2c44a6\": container with ID starting with ff75bf561b520fdedc88903cf0cf86722f0d0d76108f055304181dd49d2c44a6 not found: ID does not exist" containerID="ff75bf561b520fdedc88903cf0cf86722f0d0d76108f055304181dd49d2c44a6"
Oct 30 20:04:04.891661 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.891611    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff75bf561b520fdedc88903cf0cf86722f0d0d76108f055304181dd49d2c44a6"} err="failed to get container status \"ff75bf561b520fdedc88903cf0cf86722f0d0d76108f055304181dd49d2c44a6\": rpc error: code = NotFound desc = could not find container \"ff75bf561b520fdedc88903cf0cf86722f0d0d76108f055304181dd49d2c44a6\": container with ID starting with ff75bf561b520fdedc88903cf0cf86722f0d0d76108f055304181dd49d2c44a6 not found: ID does not exist"
Oct 30 20:04:04.912624 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.912594    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"]
Oct 30 20:04:04.915444 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:04.915424    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-836d67-66f997457c-46vmp"]
Oct 30 20:04:05.202611 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:05.202523    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="224a8d28-e274-485f-a31f-0b001e07a764" path="/var/lib/kubelet/pods/224a8d28-e274-485f-a31f-0b001e07a764/volumes"
Oct 30 20:04:06.566429 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.566396    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"]
Oct 30 20:04:06.567486 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.567465    6030 cpu_manager.go:414] "containerMap: removing container" podUID="224a8d28-e274-485f-a31f-0b001e07a764" containerName="maas-api"
Oct 30 20:04:06.567542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.567490    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="224a8d28-e274-485f-a31f-0b001e07a764" containerName="maas-api"
Oct 30 20:04:06.567542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.567508    6030 cpu_manager.go:414] "containerMap: removing container" podUID="73882a2e-adf8-4aaa-9da6-49d5e8917248" containerName="payload-processing"
Oct 30 20:04:06.567542 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.567516    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73882a2e-adf8-4aaa-9da6-49d5e8917248" containerName="payload-processing"
Oct 30 20:04:06.567638 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.567545    6030 cpu_manager.go:414] "containerMap: removing container" podUID="e1682349-9425-4aba-a4b6-01bbe7427c61" containerName="payload-pre-processing"
Oct 30 20:04:06.567638 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.567554    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1682349-9425-4aba-a4b6-01bbe7427c61" containerName="payload-pre-processing"
Oct 30 20:04:06.567757 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.567746    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="224a8d28-e274-485f-a31f-0b001e07a764" containerName="maas-api"
Oct 30 20:04:06.567792 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.567764    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="73882a2e-adf8-4aaa-9da6-49d5e8917248" containerName="payload-processing"
Oct 30 20:04:06.567792 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.567779    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1682349-9425-4aba-a4b6-01bbe7427c61" containerName="payload-pre-processing"
Oct 30 20:04:06.573702 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.573670    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:04:06.583231 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.583206    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-2a2286-serving-cert\""
Oct 30 20:04:06.586323 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.586300    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"]
Oct 30 20:04:06.661988 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.661952    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5259686e-a598-48b6-8edc-221bcfd661e7-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp\" (UID: \"5259686e-a598-48b6-8edc-221bcfd661e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:04:06.662174 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.662127    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5259686e-a598-48b6-8edc-221bcfd661e7-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp\" (UID: \"5259686e-a598-48b6-8edc-221bcfd661e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:04:06.662224 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.662192    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgmtz\" (UniqueName: \"kubernetes.io/projected/5259686e-a598-48b6-8edc-221bcfd661e7-kube-api-access-lgmtz\") pod \"maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp\" (UID: \"5259686e-a598-48b6-8edc-221bcfd661e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:04:06.662277 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.662258    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5259686e-a598-48b6-8edc-221bcfd661e7-metrics-certs\") pod \"maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp\" (UID: \"5259686e-a598-48b6-8edc-221bcfd661e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:04:06.763678 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.763644    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgmtz\" (UniqueName: \"kubernetes.io/projected/5259686e-a598-48b6-8edc-221bcfd661e7-kube-api-access-lgmtz\") pod \"maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp\" (UID: \"5259686e-a598-48b6-8edc-221bcfd661e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:04:06.763835 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.763685    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5259686e-a598-48b6-8edc-221bcfd661e7-metrics-certs\") pod \"maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp\" (UID: \"5259686e-a598-48b6-8edc-221bcfd661e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:04:06.763835 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.763753    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5259686e-a598-48b6-8edc-221bcfd661e7-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp\" (UID: \"5259686e-a598-48b6-8edc-221bcfd661e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:04:06.763975 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.763849    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5259686e-a598-48b6-8edc-221bcfd661e7-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp\" (UID: \"5259686e-a598-48b6-8edc-221bcfd661e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:04:06.764661 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.764559    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5259686e-a598-48b6-8edc-221bcfd661e7-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp\" (UID: \"5259686e-a598-48b6-8edc-221bcfd661e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:04:06.766529 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.766510    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5259686e-a598-48b6-8edc-221bcfd661e7-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp\" (UID: \"5259686e-a598-48b6-8edc-221bcfd661e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:04:06.766712 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.766690    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5259686e-a598-48b6-8edc-221bcfd661e7-metrics-certs\") pod \"maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp\" (UID: \"5259686e-a598-48b6-8edc-221bcfd661e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:04:06.780433 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.780404    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgmtz\" (UniqueName: \"kubernetes.io/projected/5259686e-a598-48b6-8edc-221bcfd661e7-kube-api-access-lgmtz\") pod \"maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp\" (UID: \"5259686e-a598-48b6-8edc-221bcfd661e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:04:06.898083 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:06.897808    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:04:07.241912 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:07.241880    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"]
Oct 30 20:04:07.244203 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:04:07.244159    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5259686e_a598_48b6_8edc_221bcfd661e7.slice/crio-41eacdd9220299ae05f01943ea5d8e465f526eef29abd02c1aa4f4a9d23b1257 WatchSource:0}: Error finding container 41eacdd9220299ae05f01943ea5d8e465f526eef29abd02c1aa4f4a9d23b1257: Status 404 returned error can't find the container with id 41eacdd9220299ae05f01943ea5d8e465f526eef29abd02c1aa4f4a9d23b1257
Oct 30 20:04:07.935964 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:07.935922    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp" event={"ID":"5259686e-a598-48b6-8edc-221bcfd661e7","Type":"ContainerStarted","Data":"2af03603be87722f3f97373b022f3361f937d9145adf6027aff20028c54a8339"}
Oct 30 20:04:07.936512 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:07.935986    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp" event={"ID":"5259686e-a598-48b6-8edc-221bcfd661e7","Type":"ContainerStarted","Data":"41eacdd9220299ae05f01943ea5d8e465f526eef29abd02c1aa4f4a9d23b1257"}
Oct 30 20:04:07.936512 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:07.936099    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:04:07.964871 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:07.964815    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp" podStartSLOduration=1.46421155 podStartE2EDuration="1.964801378s" totalImagesPullingTime="500.589828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:07.962035222 +0000 UTC m=+1926.877642091" watchObservedRunningTime="2026-10-30 20:04:07.964801378 +0000 UTC m=+1926.880408234"
Oct 30 20:04:13.948306 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:13.948274    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:04:34.300941 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:04:34.300896    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:04:42.863845 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:42.863808    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"]
Oct 30 20:04:42.868560 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:42.868539    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:04:42.892508 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:42.892473    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-517027-serving-cert\""
Oct 30 20:04:42.907248 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:42.907221    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"]
Oct 30 20:04:42.987107 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:42.987079    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86138fcf-3364-46db-ab0b-9c327868687a-metrics-certs\") pod \"maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l\" (UID: \"86138fcf-3364-46db-ab0b-9c327868687a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:04:42.987278 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:42.987150    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs655\" (UniqueName: \"kubernetes.io/projected/86138fcf-3364-46db-ab0b-9c327868687a-kube-api-access-hs655\") pod \"maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l\" (UID: \"86138fcf-3364-46db-ab0b-9c327868687a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:04:42.987338 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:42.987312    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86138fcf-3364-46db-ab0b-9c327868687a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l\" (UID: \"86138fcf-3364-46db-ab0b-9c327868687a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:04:42.987452 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:42.987435    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86138fcf-3364-46db-ab0b-9c327868687a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l\" (UID: \"86138fcf-3364-46db-ab0b-9c327868687a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:04:43.100516 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:43.100480    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86138fcf-3364-46db-ab0b-9c327868687a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l\" (UID: \"86138fcf-3364-46db-ab0b-9c327868687a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:04:43.100873 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:43.100843    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86138fcf-3364-46db-ab0b-9c327868687a-metrics-certs\") pod \"maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l\" (UID: \"86138fcf-3364-46db-ab0b-9c327868687a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:04:43.101183 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:43.101162    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs655\" (UniqueName: \"kubernetes.io/projected/86138fcf-3364-46db-ab0b-9c327868687a-kube-api-access-hs655\") pod \"maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l\" (UID: \"86138fcf-3364-46db-ab0b-9c327868687a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:04:43.102753 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:43.102722    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86138fcf-3364-46db-ab0b-9c327868687a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l\" (UID: \"86138fcf-3364-46db-ab0b-9c327868687a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:04:43.104724 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:43.104701    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86138fcf-3364-46db-ab0b-9c327868687a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l\" (UID: \"86138fcf-3364-46db-ab0b-9c327868687a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:04:43.107759 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:43.107737    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86138fcf-3364-46db-ab0b-9c327868687a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l\" (UID: \"86138fcf-3364-46db-ab0b-9c327868687a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:04:43.109613 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:43.109575    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86138fcf-3364-46db-ab0b-9c327868687a-metrics-certs\") pod \"maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l\" (UID: \"86138fcf-3364-46db-ab0b-9c327868687a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:04:43.122862 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:43.122794    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs655\" (UniqueName: \"kubernetes.io/projected/86138fcf-3364-46db-ab0b-9c327868687a-kube-api-access-hs655\") pod \"maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l\" (UID: \"86138fcf-3364-46db-ab0b-9c327868687a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:04:43.187729 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:43.187697    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:04:43.475636 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:43.475607    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"]
Oct 30 20:04:43.476512 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:04:43.476485    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86138fcf_3364_46db_ab0b_9c327868687a.slice/crio-188e9e60f815b1e2bebaa2e5e4b7f106e475ab865d40c0b729479597d0516bc9 WatchSource:0}: Error finding container 188e9e60f815b1e2bebaa2e5e4b7f106e475ab865d40c0b729479597d0516bc9: Status 404 returned error can't find the container with id 188e9e60f815b1e2bebaa2e5e4b7f106e475ab865d40c0b729479597d0516bc9
Oct 30 20:04:44.374472 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:44.374438    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l" event={"ID":"86138fcf-3364-46db-ab0b-9c327868687a","Type":"ContainerStarted","Data":"0aa922e28d25a40894dcbeb388488fbb4353eb609099f682f6fac4918361c838"}
Oct 30 20:04:44.374940 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:44.374504    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:04:44.374940 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:44.374524    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l" event={"ID":"86138fcf-3364-46db-ab0b-9c327868687a","Type":"ContainerStarted","Data":"188e9e60f815b1e2bebaa2e5e4b7f106e475ab865d40c0b729479597d0516bc9"}
Oct 30 20:04:44.405435 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:44.405391    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l" podStartSLOduration=1.941412323 podStartE2EDuration="2.405376169s" totalImagesPullingTime="463.963846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:44.403644506 +0000 UTC m=+1963.319251362" watchObservedRunningTime="2026-10-30 20:04:44.405376169 +0000 UTC m=+1963.320983025"
Oct 30 20:04:50.386307 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:04:50.386278    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:05:04.362693 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:05:04.362652    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:05:17.116102 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:17.116063    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"]
Oct 30 20:05:17.126966 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:17.126938    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"
Oct 30 20:05:17.142250 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:17.142216    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"]
Oct 30 20:05:17.153317 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:17.153288    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-5f9b3c\""
Oct 30 20:05:17.250068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:17.222084    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-5f9b3c-dockercfg-g4dlw\""
Oct 30 20:05:17.309032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:17.308988    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35eee6aa-cd49-4b82-80d2-b680e545d1a4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb\" (UID: \"35eee6aa-cd49-4b82-80d2-b680e545d1a4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"
Oct 30 20:05:17.309536 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:17.309514    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj2kl\" (UniqueName: \"kubernetes.io/projected/35eee6aa-cd49-4b82-80d2-b680e545d1a4-kube-api-access-gj2kl\") pod \"payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb\" (UID: \"35eee6aa-cd49-4b82-80d2-b680e545d1a4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"
Oct 30 20:05:17.411358 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:17.411324    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35eee6aa-cd49-4b82-80d2-b680e545d1a4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb\" (UID: \"35eee6aa-cd49-4b82-80d2-b680e545d1a4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"
Oct 30 20:05:17.411573 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:17.411554    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj2kl\" (UniqueName: \"kubernetes.io/projected/35eee6aa-cd49-4b82-80d2-b680e545d1a4-kube-api-access-gj2kl\") pod \"payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb\" (UID: \"35eee6aa-cd49-4b82-80d2-b680e545d1a4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"
Oct 30 20:05:17.412091 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:17.412073    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35eee6aa-cd49-4b82-80d2-b680e545d1a4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb\" (UID: \"35eee6aa-cd49-4b82-80d2-b680e545d1a4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"
Oct 30 20:05:17.445189 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:17.445163    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj2kl\" (UniqueName: \"kubernetes.io/projected/35eee6aa-cd49-4b82-80d2-b680e545d1a4-kube-api-access-gj2kl\") pod \"payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb\" (UID: \"35eee6aa-cd49-4b82-80d2-b680e545d1a4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"
Oct 30 20:05:17.457673 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:17.457648    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"
Oct 30 20:05:17.747347 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:17.747321    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"]
Oct 30 20:05:17.748204 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:05:17.748175    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35eee6aa_cd49_4b82_80d2_b680e545d1a4.slice/crio-0143c39675e17513ceaad17ac3882efe6fb718e059154613b453e5082c0dfa2e WatchSource:0}: Error finding container 0143c39675e17513ceaad17ac3882efe6fb718e059154613b453e5082c0dfa2e: Status 404 returned error can't find the container with id 0143c39675e17513ceaad17ac3882efe6fb718e059154613b453e5082c0dfa2e
Oct 30 20:05:18.735787 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:18.735742    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb" event={"ID":"35eee6aa-cd49-4b82-80d2-b680e545d1a4","Type":"ContainerStarted","Data":"cb4cae5c7f74a6250ce7359f40260f5a191b793ead1e010071924262bfd2b69d"}
Oct 30 20:05:18.736222 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:18.735817    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb" event={"ID":"35eee6aa-cd49-4b82-80d2-b680e545d1a4","Type":"ContainerStarted","Data":"0143c39675e17513ceaad17ac3882efe6fb718e059154613b453e5082c0dfa2e"}
Oct 30 20:05:18.736222 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:18.735965    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"
Oct 30 20:05:18.758957 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:18.758911    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb" podStartSLOduration=1.758897326 podStartE2EDuration="1.758897326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:18.757111347 +0000 UTC m=+1997.672718204" watchObservedRunningTime="2026-10-30 20:05:18.758897326 +0000 UTC m=+1997.674504181"
Oct 30 20:05:29.752555 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:29.752521    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"
Oct 30 20:05:32.085132 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.085098    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"]
Oct 30 20:05:32.090018 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.089985    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:32.105556 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.105532    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"agc-cred-331482ea82ec\""
Oct 30 20:05:32.106033 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.106015    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"kube-root-ca.crt\""
Oct 30 20:05:32.106575 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.106553    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"payload-processing-external-model-plugins-e2e-extmod-a-222480\""
Oct 30 20:05:32.106716 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.106582    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"routing-overlay\""
Oct 30 20:05:32.106716 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.106604    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"openshift-service-ca.crt\""
Oct 30 20:05:32.107828 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.107805    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"]
Oct 30 20:05:32.186204 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.186173    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h59gd\" (UniqueName: \"kubernetes.io/projected/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-kube-api-access-h59gd\") pod \"payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6\" (UID: \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:32.186366 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.186236    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6\" (UID: \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:32.186451 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.186365    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6\" (UID: \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:32.186451 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.186415    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6\" (UID: \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:32.287533 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.287493    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h59gd\" (UniqueName: \"kubernetes.io/projected/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-kube-api-access-h59gd\") pod \"payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6\" (UID: \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:32.287717 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.287547    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6\" (UID: \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:32.287717 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.287581    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6\" (UID: \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:32.287717 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.287609    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6\" (UID: \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:32.288304 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.288279    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6\" (UID: \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:32.288421 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.288360    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6\" (UID: \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:32.290187 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.290165    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6\" (UID: \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:32.303811 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.303785    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h59gd\" (UniqueName: \"kubernetes.io/projected/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-kube-api-access-h59gd\") pod \"payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6\" (UID: \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:32.432312 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.432276    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:32.859058 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:32.859018    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"]
Oct 30 20:05:32.861320 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:05:32.861296    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80f0ff2_72f4_4714_9eb4_f1ffc297e505.slice/crio-f85caa9a1a53746017c53af2adc321931a5733455971ada6843400c387b96709 WatchSource:0}: Error finding container f85caa9a1a53746017c53af2adc321931a5733455971ada6843400c387b96709: Status 404 returned error can't find the container with id f85caa9a1a53746017c53af2adc321931a5733455971ada6843400c387b96709
Oct 30 20:05:33.850223 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:33.850189    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6" event={"ID":"d80f0ff2-72f4-4714-9eb4-f1ffc297e505","Type":"ContainerStarted","Data":"f21e8df02fc1bd951cd73743f71d5745027d36c602db5bcd1ccc9c897989f1be"}
Oct 30 20:05:33.850717 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:33.850243    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6" event={"ID":"d80f0ff2-72f4-4714-9eb4-f1ffc297e505","Type":"ContainerStarted","Data":"f85caa9a1a53746017c53af2adc321931a5733455971ada6843400c387b96709"}
Oct 30 20:05:33.850717 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:33.850290    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:33.873979 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:33.873922    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6" podStartSLOduration=1.873906941 podStartE2EDuration="1.873906941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:33.872205551 +0000 UTC m=+2012.787812411" watchObservedRunningTime="2026-10-30 20:05:33.873906941 +0000 UTC m=+2012.789513798"
Oct 30 20:05:34.310584 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:05:34.310546    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:05:38.673798 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:38.673352    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"]
Oct 30 20:05:38.679861 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:38.679833    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:43.673824 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:43.673779    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6" podUID="d80f0ff2-72f4-4714-9eb4-f1ffc297e505" containerName="payload-processing" containerID="cri-o://f21e8df02fc1bd951cd73743f71d5745027d36c602db5bcd1ccc9c897989f1be" gracePeriod=25
Oct 30 20:05:43.957199 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:43.957166    6030 generic.go:356] "Generic (PLEG): container finished" podID="d80f0ff2-72f4-4714-9eb4-f1ffc297e505" containerID="f21e8df02fc1bd951cd73743f71d5745027d36c602db5bcd1ccc9c897989f1be" exitCode=0
Oct 30 20:05:43.957349 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:43.957238    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6" event={"ID":"d80f0ff2-72f4-4714-9eb4-f1ffc297e505","Type":"ContainerDied","Data":"f21e8df02fc1bd951cd73743f71d5745027d36c602db5bcd1ccc9c897989f1be"}
Oct 30 20:05:43.992890 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:43.992861    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:44.086945 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:44.086909    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-kube-api-access-h59gd\" (UniqueName: \"kubernetes.io/projected/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-kube-api-access-h59gd\") pod \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\" (UID: \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\") "
Oct 30 20:05:44.087157 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:44.087070    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-routing-overlay-volume\") pod \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\" (UID: \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\") "
Oct 30 20:05:44.087157 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:44.087118    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-provider-credentials\") pod \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\" (UID: \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\") "
Oct 30 20:05:44.087318 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:44.087289    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-plugins-config-volume\") pod \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\" (UID: \"d80f0ff2-72f4-4714-9eb4-f1ffc297e505\") "
Oct 30 20:05:44.087527 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:44.087427    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-routing-overlay-volume" pod "d80f0ff2-72f4-4714-9eb4-f1ffc297e505" (UID: "d80f0ff2-72f4-4714-9eb4-f1ffc297e505"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:44.087668 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:44.087615    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-plugins-config-volume" pod "d80f0ff2-72f4-4714-9eb4-f1ffc297e505" (UID: "d80f0ff2-72f4-4714-9eb4-f1ffc297e505"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:44.087917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:44.087900    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:44.088025 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:44.087922    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:44.089778 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:44.089743    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-kube-api-access-h59gd" pod "d80f0ff2-72f4-4714-9eb4-f1ffc297e505" (UID: "d80f0ff2-72f4-4714-9eb4-f1ffc297e505"). InnerVolumeSpecName "kube-api-access-h59gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:44.089889 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:44.089834    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-provider-credentials" pod "d80f0ff2-72f4-4714-9eb4-f1ffc297e505" (UID: "d80f0ff2-72f4-4714-9eb4-f1ffc297e505"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:44.188866 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:44.188836    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h59gd\" (UniqueName: \"kubernetes.io/projected/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-kube-api-access-h59gd\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:44.188866 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:44.188866    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d80f0ff2-72f4-4714-9eb4-f1ffc297e505-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:44.967123 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:44.967091    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"
Oct 30 20:05:44.967123 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:44.967116    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6" event={"ID":"d80f0ff2-72f4-4714-9eb4-f1ffc297e505","Type":"ContainerDied","Data":"f85caa9a1a53746017c53af2adc321931a5733455971ada6843400c387b96709"}
Oct 30 20:05:44.967688 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:44.967187    6030 scope.go:122] "RemoveContainer" containerID="f21e8df02fc1bd951cd73743f71d5745027d36c602db5bcd1ccc9c897989f1be"
Oct 30 20:05:45.052183 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:45.052157    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"]
Oct 30 20:05:45.059769 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:45.059744    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-b549dcfq6"]
Oct 30 20:05:45.203700 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:05:45.203664    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d80f0ff2-72f4-4714-9eb4-f1ffc297e505" path="/var/lib/kubelet/pods/d80f0ff2-72f4-4714-9eb4-f1ffc297e505/volumes"
Oct 30 20:06:04.323284 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:06:04.323247    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:06:04.945180 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:04.945147    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"]
Oct 30 20:06:04.945888 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:04.945871    6030 cpu_manager.go:414] "containerMap: removing container" podUID="d80f0ff2-72f4-4714-9eb4-f1ffc297e505" containerName="payload-processing"
Oct 30 20:06:04.945938 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:04.945890    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d80f0ff2-72f4-4714-9eb4-f1ffc297e505" containerName="payload-processing"
Oct 30 20:06:04.946080 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:04.946064    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="d80f0ff2-72f4-4714-9eb4-f1ffc297e505" containerName="payload-processing"
Oct 30 20:06:04.951016 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:04.950996    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:04.963302 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:04.963266    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"agc-cred-9cd68f49947c\""
Oct 30 20:06:04.963519 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:04.963496    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"payload-processing-external-model-e2e--8e4491f3-dockercfg-rdsrv\""
Oct 30 20:06:04.964649 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:04.964626    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"]
Oct 30 20:06:04.965524 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:04.965503    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"routing-overlay\""
Oct 30 20:06:04.965779 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:04.965762    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"kube-root-ca.crt\""
Oct 30 20:06:04.965936 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:04.965918    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"payload-processing-external-model-plugins-e2e-extmod-a-222480\""
Oct 30 20:06:04.966264 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:04.966247    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"openshift-service-ca.crt\""
Oct 30 20:06:05.072065 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:05.068353    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf\" (UID: \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:05.072065 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:05.068447    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf\" (UID: \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:05.072065 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:05.068530    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf\" (UID: \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:05.072065 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:05.068660    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4s9t\" (UniqueName: \"kubernetes.io/projected/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-kube-api-access-l4s9t\") pod \"payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf\" (UID: \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:05.177313 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:05.172598    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf\" (UID: \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:05.177313 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:05.172688    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf\" (UID: \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:05.177313 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:05.172760    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf\" (UID: \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:05.177313 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:05.172878    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4s9t\" (UniqueName: \"kubernetes.io/projected/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-kube-api-access-l4s9t\") pod \"payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf\" (UID: \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:05.177313 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:05.174131    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf\" (UID: \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:05.177313 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:05.175015    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf\" (UID: \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:05.210134 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:05.195543    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf\" (UID: \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:05.219164 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:05.211421    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4s9t\" (UniqueName: \"kubernetes.io/projected/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-kube-api-access-l4s9t\") pod \"payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf\" (UID: \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:05.269464 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:05.269430    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:05.591066 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:05.584995    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"]
Oct 30 20:06:06.237768 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:06.237725    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf" event={"ID":"d09c8d1a-a772-47dc-a1ea-95b248cf0f16","Type":"ContainerStarted","Data":"c818f4b514be0475bc969088c0d38e148f56a846c920400c69e23d0997edd1dc"}
Oct 30 20:06:06.237988 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:06.237790    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:06.237988 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:06.237813    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf" event={"ID":"d09c8d1a-a772-47dc-a1ea-95b248cf0f16","Type":"ContainerStarted","Data":"f27e813e8da4d863648f5f491d273d9f050c297447b3ddc5cda3582092f8f5b4"}
Oct 30 20:06:06.260440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:06.260336    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf" podStartSLOduration=2.260319066 podStartE2EDuration="2.260319066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:06.259968915 +0000 UTC m=+2045.175575771" watchObservedRunningTime="2026-10-30 20:06:06.260319066 +0000 UTC m=+2045.175925922"
Oct 30 20:06:11.019995 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:11.019963    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"]
Oct 30 20:06:11.023599 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:11.023572    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:16.020970 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.020931    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf" podUID="d09c8d1a-a772-47dc-a1ea-95b248cf0f16" containerName="payload-processing" containerID="cri-o://c818f4b514be0475bc969088c0d38e148f56a846c920400c69e23d0997edd1dc" gracePeriod=25
Oct 30 20:06:16.302125 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.302101    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:16.343135 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.343098    6030 generic.go:356] "Generic (PLEG): container finished" podID="d09c8d1a-a772-47dc-a1ea-95b248cf0f16" containerID="c818f4b514be0475bc969088c0d38e148f56a846c920400c69e23d0997edd1dc" exitCode=0
Oct 30 20:06:16.343399 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.343173    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"
Oct 30 20:06:16.343587 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.343209    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf" event={"ID":"d09c8d1a-a772-47dc-a1ea-95b248cf0f16","Type":"ContainerDied","Data":"c818f4b514be0475bc969088c0d38e148f56a846c920400c69e23d0997edd1dc"}
Oct 30 20:06:16.343719 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.343632    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf" event={"ID":"d09c8d1a-a772-47dc-a1ea-95b248cf0f16","Type":"ContainerDied","Data":"f27e813e8da4d863648f5f491d273d9f050c297447b3ddc5cda3582092f8f5b4"}
Oct 30 20:06:16.343719 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.343670    6030 scope.go:122] "RemoveContainer" containerID="c818f4b514be0475bc969088c0d38e148f56a846c920400c69e23d0997edd1dc"
Oct 30 20:06:16.355448 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.355425    6030 scope.go:122] "RemoveContainer" containerID="c818f4b514be0475bc969088c0d38e148f56a846c920400c69e23d0997edd1dc"
Oct 30 20:06:16.355784 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:06:16.355760    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c818f4b514be0475bc969088c0d38e148f56a846c920400c69e23d0997edd1dc\": container with ID starting with c818f4b514be0475bc969088c0d38e148f56a846c920400c69e23d0997edd1dc not found: ID does not exist" containerID="c818f4b514be0475bc969088c0d38e148f56a846c920400c69e23d0997edd1dc"
Oct 30 20:06:16.355871 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.355795    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c818f4b514be0475bc969088c0d38e148f56a846c920400c69e23d0997edd1dc"} err="failed to get container status \"c818f4b514be0475bc969088c0d38e148f56a846c920400c69e23d0997edd1dc\": rpc error: code = NotFound desc = could not find container \"c818f4b514be0475bc969088c0d38e148f56a846c920400c69e23d0997edd1dc\": container with ID starting with c818f4b514be0475bc969088c0d38e148f56a846c920400c69e23d0997edd1dc not found: ID does not exist"
Oct 30 20:06:16.396803 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.396780    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-kube-api-access-l4s9t\" (UniqueName: \"kubernetes.io/projected/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-kube-api-access-l4s9t\") pod \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\" (UID: \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\") "
Oct 30 20:06:16.396955 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.396944    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-plugins-config-volume\") pod \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\" (UID: \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\") "
Oct 30 20:06:16.397003 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.396970    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-provider-credentials\") pod \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\" (UID: \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\") "
Oct 30 20:06:16.397003 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.396992    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-routing-overlay-volume\") pod \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\" (UID: \"d09c8d1a-a772-47dc-a1ea-95b248cf0f16\") "
Oct 30 20:06:16.397384 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.397358    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-plugins-config-volume" pod "d09c8d1a-a772-47dc-a1ea-95b248cf0f16" (UID: "d09c8d1a-a772-47dc-a1ea-95b248cf0f16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:16.397490 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.397404    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-routing-overlay-volume" pod "d09c8d1a-a772-47dc-a1ea-95b248cf0f16" (UID: "d09c8d1a-a772-47dc-a1ea-95b248cf0f16"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:16.397613 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.397599    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:16.397655 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.397616    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:16.399098 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.399080    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-kube-api-access-l4s9t" pod "d09c8d1a-a772-47dc-a1ea-95b248cf0f16" (UID: "d09c8d1a-a772-47dc-a1ea-95b248cf0f16"). InnerVolumeSpecName "kube-api-access-l4s9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:16.399169 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.399127    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-provider-credentials" pod "d09c8d1a-a772-47dc-a1ea-95b248cf0f16" (UID: "d09c8d1a-a772-47dc-a1ea-95b248cf0f16"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:16.498683 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.498645    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:16.498683 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.498671    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4s9t\" (UniqueName: \"kubernetes.io/projected/d09c8d1a-a772-47dc-a1ea-95b248cf0f16-kube-api-access-l4s9t\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:16.673989 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.673957    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"]
Oct 30 20:06:16.676911 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:16.676885    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-55bbjgpmf"]
Oct 30 20:06:17.216366 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:17.216334    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d09c8d1a-a772-47dc-a1ea-95b248cf0f16" path="/var/lib/kubelet/pods/d09c8d1a-a772-47dc-a1ea-95b248cf0f16/volumes"
Oct 30 20:06:23.056481 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.056445    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"]
Oct 30 20:06:23.057371 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.057343    6030 cpu_manager.go:414] "containerMap: removing container" podUID="d09c8d1a-a772-47dc-a1ea-95b248cf0f16" containerName="payload-processing"
Oct 30 20:06:23.057371 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.057367    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d09c8d1a-a772-47dc-a1ea-95b248cf0f16" containerName="payload-processing"
Oct 30 20:06:23.057532 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.057486    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="d09c8d1a-a772-47dc-a1ea-95b248cf0f16" containerName="payload-processing"
Oct 30 20:06:23.061205 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.061187    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.071586 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.071563    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-2a2286-openshift-default-dockercfg-grjl9\""
Oct 30 20:06:23.078185 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.078160    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"]
Oct 30 20:06:23.130148 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.130103    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-credential-socket\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.130307 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.130157    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdeef59a-a6c5-49c9-a081-8d374f01166d-istiod-ca-cert\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.130307 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.130270    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-podinfo\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.130385 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.130335    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-workload-socket\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.130385 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.130364    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-data\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.130452 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.130385    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-token\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.130452 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.130410    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44t7d\" (UniqueName: \"kubernetes.io/projected/cdeef59a-a6c5-49c9-a081-8d374f01166d-kube-api-access-44t7d\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.130515 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.130500    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-workload-certs\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.130685 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.130666    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-envoy\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.232295 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.232264    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-envoy\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.232478 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.232301    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-credential-socket\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.232478 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.232330    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdeef59a-a6c5-49c9-a081-8d374f01166d-istiod-ca-cert\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.232478 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.232363    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-podinfo\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.232478 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.232406    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-workload-socket\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.232478 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.232438    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-data\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.232478 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.232468    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-token\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.232796 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.232495    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44t7d\" (UniqueName: \"kubernetes.io/projected/cdeef59a-a6c5-49c9-a081-8d374f01166d-kube-api-access-44t7d\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.232796 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.232607    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-workload-certs\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.232796 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.232756    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-credential-socket\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.232974 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.232791    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-data\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.233110 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.233085    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-workload-socket\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.233230 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.233203    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdeef59a-a6c5-49c9-a081-8d374f01166d-istiod-ca-cert\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.233468 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.233219    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-workload-certs\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.234882 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.234860    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-envoy\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.235121 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.235102    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-podinfo\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.247999 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.247972    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-token\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.248528 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.248509    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44t7d\" (UniqueName: \"kubernetes.io/projected/cdeef59a-a6c5-49c9-a081-8d374f01166d-kube-api-access-44t7d\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.384672 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.384640    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:23.674835 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.674795    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"]
Oct 30 20:06:23.675411 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:06:23.675384    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdeef59a_a6c5_49c9_a081_8d374f01166d.slice/crio-476cba67a4e0ef2606a07f2259f3f98e756dabba06396f9828c84cebaa813520 WatchSource:0}: Error finding container 476cba67a4e0ef2606a07f2259f3f98e756dabba06396f9828c84cebaa813520: Status 404 returned error can't find the container with id 476cba67a4e0ef2606a07f2259f3f98e756dabba06396f9828c84cebaa813520
Oct 30 20:06:23.678562 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.678523    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:06:23.678657 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.678593    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:06:23.678657 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:23.678622    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:06:24.440069 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:24.438868    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk" event={"ID":"cdeef59a-a6c5-49c9-a081-8d374f01166d","Type":"ContainerStarted","Data":"be7d269465d17b685543a6f5b5b4e0667c15cc99a0429dc276ce7e4f6881c5bb"}
Oct 30 20:06:24.440069 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:24.438941    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk" event={"ID":"cdeef59a-a6c5-49c9-a081-8d374f01166d","Type":"ContainerStarted","Data":"476cba67a4e0ef2606a07f2259f3f98e756dabba06396f9828c84cebaa813520"}
Oct 30 20:06:24.487070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:24.484856    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk" podStartSLOduration=1.4848376220000001 podStartE2EDuration="1.484837622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:24.477558448 +0000 UTC m=+2063.393165304" watchObservedRunningTime="2026-10-30 20:06:24.484837622 +0000 UTC m=+2063.400444478"
Oct 30 20:06:25.394071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:25.393254    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:25.394301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:25.394163    6030 patch_prober.go:29] interesting pod/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 20:06:25.394301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:25.394231    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk" podUID="cdeef59a-a6c5-49c9-a081-8d374f01166d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 20:06:26.389070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:26.385712    6030 patch_prober.go:29] interesting pod/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 20:06:26.389070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:26.385787    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk" podUID="cdeef59a-a6c5-49c9-a081-8d374f01166d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 20:06:27.393498 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:27.393460    6030 patch_prober.go:29] interesting pod/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 20:06:27.402080 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:27.401583    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk" podUID="cdeef59a-a6c5-49c9-a081-8d374f01166d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 20:06:28.388434 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:28.388394    6030 patch_prober.go:29] interesting pod/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 20:06:28.389921 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:28.389887    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk" podUID="cdeef59a-a6c5-49c9-a081-8d374f01166d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 20:06:29.387071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:29.385409    6030 patch_prober.go:29] interesting pod/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 20:06:29.387071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:29.385504    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk" podUID="cdeef59a-a6c5-49c9-a081-8d374f01166d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 20:06:30.385729 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:30.385691    6030 patch_prober.go:29] interesting pod/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 20:06:30.386021 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:30.385993    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk" podUID="cdeef59a-a6c5-49c9-a081-8d374f01166d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 20:06:31.391828 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:31.391798    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:31.392298 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:31.392106    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:31.393208 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:06:31.393188    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:06:34.290430 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:06:34.290393    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:07:04.293239 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:07:04.293198    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:07:25.449075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:25.445546    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"]
Oct 30 20:07:25.449075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:25.445770    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9" podUID="409380a1-684a-4be3-9cfc-a9e72c247409" containerName="istio-proxy" containerID="cri-o://49f2cb1762d21c727d9619f77571c7832d10842769c9806268122b190c9ac855" gracePeriod=30
Oct 30 20:07:30.233240 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.233206    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:07:30.233628 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.233260    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9" podUID="409380a1-684a-4be3-9cfc-a9e72c247409" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:07:30.724521 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.724494    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 20:07:30.882259 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.878075    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-workload-socket\") pod \"409380a1-684a-4be3-9cfc-a9e72c247409\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") "
Oct 30 20:07:30.882259 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.878161    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-credential-socket\") pod \"409380a1-684a-4be3-9cfc-a9e72c247409\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") "
Oct 30 20:07:30.882259 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.878218    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409380a1-684a-4be3-9cfc-a9e72c247409-istio-token\" (UniqueName: \"kubernetes.io/projected/409380a1-684a-4be3-9cfc-a9e72c247409-istio-token\") pod \"409380a1-684a-4be3-9cfc-a9e72c247409\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") "
Oct 30 20:07:30.882259 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.878285    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-istio-data\") pod \"409380a1-684a-4be3-9cfc-a9e72c247409\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") "
Oct 30 20:07:30.882259 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.878325    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409380a1-684a-4be3-9cfc-a9e72c247409-kube-api-access-mrw29\" (UniqueName: \"kubernetes.io/projected/409380a1-684a-4be3-9cfc-a9e72c247409-kube-api-access-mrw29\") pod \"409380a1-684a-4be3-9cfc-a9e72c247409\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") "
Oct 30 20:07:30.882259 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.878400    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-istio-envoy\") pod \"409380a1-684a-4be3-9cfc-a9e72c247409\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") "
Oct 30 20:07:30.882259 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.878536    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-workload-certs\") pod \"409380a1-684a-4be3-9cfc-a9e72c247409\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") "
Oct 30 20:07:30.882259 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.878563    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/409380a1-684a-4be3-9cfc-a9e72c247409-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409380a1-684a-4be3-9cfc-a9e72c247409-istiod-ca-cert\") pod \"409380a1-684a-4be3-9cfc-a9e72c247409\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") "
Oct 30 20:07:30.882259 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.878781    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/409380a1-684a-4be3-9cfc-a9e72c247409-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409380a1-684a-4be3-9cfc-a9e72c247409-istio-podinfo\") pod \"409380a1-684a-4be3-9cfc-a9e72c247409\" (UID: \"409380a1-684a-4be3-9cfc-a9e72c247409\") "
Oct 30 20:07:30.882259 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.880664    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-istio-data" pod "409380a1-684a-4be3-9cfc-a9e72c247409" (UID: "409380a1-684a-4be3-9cfc-a9e72c247409"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:07:30.882259 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.880874    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-workload-socket" pod "409380a1-684a-4be3-9cfc-a9e72c247409" (UID: "409380a1-684a-4be3-9cfc-a9e72c247409"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:07:30.882259 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.881090    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-credential-socket" pod "409380a1-684a-4be3-9cfc-a9e72c247409" (UID: "409380a1-684a-4be3-9cfc-a9e72c247409"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:07:30.883092 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.882679    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409380a1-684a-4be3-9cfc-a9e72c247409-istiod-ca-cert" pod "409380a1-684a-4be3-9cfc-a9e72c247409" (UID: "409380a1-684a-4be3-9cfc-a9e72c247409"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:30.883092 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.882737    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-workload-certs" pod "409380a1-684a-4be3-9cfc-a9e72c247409" (UID: "409380a1-684a-4be3-9cfc-a9e72c247409"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:07:30.886529 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.886479    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409380a1-684a-4be3-9cfc-a9e72c247409-istio-token" pod "409380a1-684a-4be3-9cfc-a9e72c247409" (UID: "409380a1-684a-4be3-9cfc-a9e72c247409"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:30.918274 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.918230    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409380a1-684a-4be3-9cfc-a9e72c247409-kube-api-access-mrw29" pod "409380a1-684a-4be3-9cfc-a9e72c247409" (UID: "409380a1-684a-4be3-9cfc-a9e72c247409"). InnerVolumeSpecName "kube-api-access-mrw29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:30.918472 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.918305    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/409380a1-684a-4be3-9cfc-a9e72c247409-istio-podinfo" pod "409380a1-684a-4be3-9cfc-a9e72c247409" (UID: "409380a1-684a-4be3-9cfc-a9e72c247409"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:07:30.918472 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.918375    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-istio-envoy" pod "409380a1-684a-4be3-9cfc-a9e72c247409" (UID: "409380a1-684a-4be3-9cfc-a9e72c247409"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:07:30.980462 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.980430    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409380a1-684a-4be3-9cfc-a9e72c247409-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:30.980462 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.980457    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:30.980462 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.980466    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrw29\" (UniqueName: \"kubernetes.io/projected/409380a1-684a-4be3-9cfc-a9e72c247409-kube-api-access-mrw29\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:30.980692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.980476    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:30.980692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.980485    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:30.980692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.980493    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409380a1-684a-4be3-9cfc-a9e72c247409-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:30.980692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.980501    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409380a1-684a-4be3-9cfc-a9e72c247409-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:30.980692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.980509    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:30.980692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:30.980516    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409380a1-684a-4be3-9cfc-a9e72c247409-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:31.165063 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:31.164960    6030 generic.go:356] "Generic (PLEG): container finished" podID="409380a1-684a-4be3-9cfc-a9e72c247409" containerID="49f2cb1762d21c727d9619f77571c7832d10842769c9806268122b190c9ac855" exitCode=0
Oct 30 20:07:31.165063 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:31.165012    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"
Oct 30 20:07:31.165063 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:31.165038    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9" event={"ID":"409380a1-684a-4be3-9cfc-a9e72c247409","Type":"ContainerDied","Data":"49f2cb1762d21c727d9619f77571c7832d10842769c9806268122b190c9ac855"}
Oct 30 20:07:31.165334 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:31.165107    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9" event={"ID":"409380a1-684a-4be3-9cfc-a9e72c247409","Type":"ContainerDied","Data":"4f0068b30c28587fff01c680e2227adc202be305cab6242309bcbc78d4bde2a1"}
Oct 30 20:07:31.165334 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:31.165130    6030 scope.go:122] "RemoveContainer" containerID="49f2cb1762d21c727d9619f77571c7832d10842769c9806268122b190c9ac855"
Oct 30 20:07:31.175718 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:31.175699    6030 scope.go:122] "RemoveContainer" containerID="49f2cb1762d21c727d9619f77571c7832d10842769c9806268122b190c9ac855"
Oct 30 20:07:31.175965 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:07:31.175947    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f2cb1762d21c727d9619f77571c7832d10842769c9806268122b190c9ac855\": container with ID starting with 49f2cb1762d21c727d9619f77571c7832d10842769c9806268122b190c9ac855 not found: ID does not exist" containerID="49f2cb1762d21c727d9619f77571c7832d10842769c9806268122b190c9ac855"
Oct 30 20:07:31.176008 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:31.175974    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f2cb1762d21c727d9619f77571c7832d10842769c9806268122b190c9ac855"} err="failed to get container status \"49f2cb1762d21c727d9619f77571c7832d10842769c9806268122b190c9ac855\": rpc error: code = NotFound desc = could not find container \"49f2cb1762d21c727d9619f77571c7832d10842769c9806268122b190c9ac855\": container with ID starting with 49f2cb1762d21c727d9619f77571c7832d10842769c9806268122b190c9ac855 not found: ID does not exist"
Oct 30 20:07:31.267509 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:31.267469    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"]
Oct 30 20:07:31.268127 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:31.268096    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-89qq9"]
Oct 30 20:07:33.203332 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:33.203300    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="409380a1-684a-4be3-9cfc-a9e72c247409" path="/var/lib/kubelet/pods/409380a1-684a-4be3-9cfc-a9e72c247409/volumes"
Oct 30 20:07:34.289901 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:07:34.289861    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:07:47.146810 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.146765    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"]
Oct 30 20:07:47.147577 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.147556    6030 cpu_manager.go:414] "containerMap: removing container" podUID="409380a1-684a-4be3-9cfc-a9e72c247409" containerName="istio-proxy"
Oct 30 20:07:47.147577 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.147576    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409380a1-684a-4be3-9cfc-a9e72c247409" containerName="istio-proxy"
Oct 30 20:07:47.147741 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.147734    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="409380a1-684a-4be3-9cfc-a9e72c247409" containerName="istio-proxy"
Oct 30 20:07:47.153256 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.153229    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:07:47.160755 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.160730    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"]
Oct 30 20:07:47.175416 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.175390    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"e2e-cfl-19316d-provider-cred\""
Oct 30 20:07:47.175618 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.175441    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"agc-cred-2f53fa2ffa5e\""
Oct 30 20:07:47.175749 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.175606    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"payload-processing-external-model-e2e--babb2686-dockercfg-tc6p4\""
Oct 30 20:07:47.175837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.175745    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"openshift-service-ca.crt\""
Oct 30 20:07:47.177119 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.177093    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"routing-overlay\""
Oct 30 20:07:47.177223 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.177124    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"kube-root-ca.crt\""
Oct 30 20:07:47.177223 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.177215    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-68685b\""
Oct 30 20:07:47.188086 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.188033    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a3cf286-3aa5-47a2-b87a-a698306d041d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n\" (UID: \"4a3cf286-3aa5-47a2-b87a-a698306d041d\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:07:47.188184 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.188151    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a3cf286-3aa5-47a2-b87a-a698306d041d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n\" (UID: \"4a3cf286-3aa5-47a2-b87a-a698306d041d\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:07:47.188237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.188192    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3cf286-3aa5-47a2-b87a-a698306d041d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n\" (UID: \"4a3cf286-3aa5-47a2-b87a-a698306d041d\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:07:47.188365 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.188343    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gftt\" (UniqueName: \"kubernetes.io/projected/4a3cf286-3aa5-47a2-b87a-a698306d041d-kube-api-access-7gftt\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n\" (UID: \"4a3cf286-3aa5-47a2-b87a-a698306d041d\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:07:47.289536 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.289497    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a3cf286-3aa5-47a2-b87a-a698306d041d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n\" (UID: \"4a3cf286-3aa5-47a2-b87a-a698306d041d\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:07:47.289743 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.289553    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a3cf286-3aa5-47a2-b87a-a698306d041d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n\" (UID: \"4a3cf286-3aa5-47a2-b87a-a698306d041d\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:07:47.289743 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.289589    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3cf286-3aa5-47a2-b87a-a698306d041d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n\" (UID: \"4a3cf286-3aa5-47a2-b87a-a698306d041d\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:07:47.289915 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.289765    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gftt\" (UniqueName: \"kubernetes.io/projected/4a3cf286-3aa5-47a2-b87a-a698306d041d-kube-api-access-7gftt\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n\" (UID: \"4a3cf286-3aa5-47a2-b87a-a698306d041d\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:07:47.290300 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.290276    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a3cf286-3aa5-47a2-b87a-a698306d041d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n\" (UID: \"4a3cf286-3aa5-47a2-b87a-a698306d041d\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:07:47.290410 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.290387    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3cf286-3aa5-47a2-b87a-a698306d041d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n\" (UID: \"4a3cf286-3aa5-47a2-b87a-a698306d041d\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:07:47.292302 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.292283    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a3cf286-3aa5-47a2-b87a-a698306d041d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n\" (UID: \"4a3cf286-3aa5-47a2-b87a-a698306d041d\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:07:47.315963 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.315937    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gftt\" (UniqueName: \"kubernetes.io/projected/4a3cf286-3aa5-47a2-b87a-a698306d041d-kube-api-access-7gftt\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n\" (UID: \"4a3cf286-3aa5-47a2-b87a-a698306d041d\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:07:47.478012 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.477929    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:07:47.773244 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:47.773203    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"]
Oct 30 20:07:47.774549 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:07:47.774519    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3cf286_3aa5_47a2_b87a_a698306d041d.slice/crio-1ee7d1089c24141c5ff0238138e4aa47467720e9dfe1bc09b8dc7cb864a16b6e WatchSource:0}: Error finding container 1ee7d1089c24141c5ff0238138e4aa47467720e9dfe1bc09b8dc7cb864a16b6e: Status 404 returned error can't find the container with id 1ee7d1089c24141c5ff0238138e4aa47467720e9dfe1bc09b8dc7cb864a16b6e
Oct 30 20:07:48.403448 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:48.403404    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n" event={"ID":"4a3cf286-3aa5-47a2-b87a-a698306d041d","Type":"ContainerStarted","Data":"359bc5098d81695eebb5c5a4a6fea9a60538eb81d8bb38dc83523a86d3c8a7cd"}
Oct 30 20:07:48.403903 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:48.403500    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n" event={"ID":"4a3cf286-3aa5-47a2-b87a-a698306d041d","Type":"ContainerStarted","Data":"1ee7d1089c24141c5ff0238138e4aa47467720e9dfe1bc09b8dc7cb864a16b6e"}
Oct 30 20:07:48.403903 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:48.403643    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:07:48.441017 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:48.440966    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n" podStartSLOduration=1.44095294 podStartE2EDuration="1.44095294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:48.439596928 +0000 UTC m=+2147.355203805" watchObservedRunningTime="2026-10-30 20:07:48.44095294 +0000 UTC m=+2147.356559796"
Oct 30 20:07:59.413429 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:07:59.413397    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:08:04.290738 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:08:04.290700    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:08:29.814082 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:29.814022    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"]
Oct 30 20:08:29.814726 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:29.814703    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"]
Oct 30 20:08:29.815032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:29.815004    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9" podUID="505b8b25-660e-432f-87b0-f359078e9f2a" containerName="istio-proxy" containerID="cri-o://e715322eac72ff1c96c639b90f4fb91fe32c3e0a14647cd05480cdcf30e7ab1a" gracePeriod=30
Oct 30 20:08:29.815146 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:29.815074    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l" podUID="57e688d9-46c5-47a1-8150-de9e8e024c49" containerName="istio-proxy" containerID="cri-o://ee2d2aa112d61de791a8e6e250487c0daf1922773f771b8bbda09cd3f7751e92" gracePeriod=30
Oct 30 20:08:34.290014 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:08:34.289976    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:08:35.175570 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.175535    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 20:08:35.209645 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.209622    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 20:08:35.378199 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.376723    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57e688d9-46c5-47a1-8150-de9e8e024c49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e688d9-46c5-47a1-8150-de9e8e024c49-istiod-ca-cert\") pod \"57e688d9-46c5-47a1-8150-de9e8e024c49\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") "
Oct 30 20:08:35.378199 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.376809    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/505b8b25-660e-432f-87b0-f359078e9f2a-istio-token\" (UniqueName: \"kubernetes.io/projected/505b8b25-660e-432f-87b0-f359078e9f2a-istio-token\") pod \"505b8b25-660e-432f-87b0-f359078e9f2a\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") "
Oct 30 20:08:35.378199 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.376872    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-workload-certs\") pod \"505b8b25-660e-432f-87b0-f359078e9f2a\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") "
Oct 30 20:08:35.378199 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.376901    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-workload-certs\") pod \"57e688d9-46c5-47a1-8150-de9e8e024c49\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") "
Oct 30 20:08:35.378199 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.376929    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-podinfo\") pod \"57e688d9-46c5-47a1-8150-de9e8e024c49\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") "
Oct 30 20:08:35.378199 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.377009    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/505b8b25-660e-432f-87b0-f359078e9f2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505b8b25-660e-432f-87b0-f359078e9f2a-istio-podinfo\") pod \"505b8b25-660e-432f-87b0-f359078e9f2a\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") "
Oct 30 20:08:35.378199 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.377081    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-workload-socket\") pod \"57e688d9-46c5-47a1-8150-de9e8e024c49\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") "
Oct 30 20:08:35.378199 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.377126    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-data\") pod \"57e688d9-46c5-47a1-8150-de9e8e024c49\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") "
Oct 30 20:08:35.378199 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.377154    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-credential-socket\") pod \"505b8b25-660e-432f-87b0-f359078e9f2a\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") "
Oct 30 20:08:35.378199 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.377239    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-istio-envoy\") pod \"505b8b25-660e-432f-87b0-f359078e9f2a\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") "
Oct 30 20:08:35.378199 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.377585    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-data" pod "57e688d9-46c5-47a1-8150-de9e8e024c49" (UID: "57e688d9-46c5-47a1-8150-de9e8e024c49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:35.379430 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.378240    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-workload-socket" pod "57e688d9-46c5-47a1-8150-de9e8e024c49" (UID: "57e688d9-46c5-47a1-8150-de9e8e024c49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:35.379430 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.378642    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-workload-certs" pod "505b8b25-660e-432f-87b0-f359078e9f2a" (UID: "505b8b25-660e-432f-87b0-f359078e9f2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:35.379430 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.378820    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-workload-certs" pod "57e688d9-46c5-47a1-8150-de9e8e024c49" (UID: "57e688d9-46c5-47a1-8150-de9e8e024c49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:35.379430 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.378886    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-envoy\") pod \"57e688d9-46c5-47a1-8150-de9e8e024c49\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") "
Oct 30 20:08:35.379430 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.378938    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-credential-socket\") pod \"57e688d9-46c5-47a1-8150-de9e8e024c49\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") "
Oct 30 20:08:35.379430 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.378993    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/505b8b25-660e-432f-87b0-f359078e9f2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505b8b25-660e-432f-87b0-f359078e9f2a-istiod-ca-cert\") pod \"505b8b25-660e-432f-87b0-f359078e9f2a\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") "
Oct 30 20:08:35.379430 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.379027    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-istio-data\") pod \"505b8b25-660e-432f-87b0-f359078e9f2a\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") "
Oct 30 20:08:35.379430 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.379096    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-token\" (UniqueName: \"kubernetes.io/projected/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-token\") pod \"57e688d9-46c5-47a1-8150-de9e8e024c49\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") "
Oct 30 20:08:35.379430 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.379137    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/505b8b25-660e-432f-87b0-f359078e9f2a-kube-api-access-cmwmz\" (UniqueName: \"kubernetes.io/projected/505b8b25-660e-432f-87b0-f359078e9f2a-kube-api-access-cmwmz\") pod \"505b8b25-660e-432f-87b0-f359078e9f2a\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") "
Oct 30 20:08:35.379430 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.379212    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e688d9-46c5-47a1-8150-de9e8e024c49-kube-api-access-cvz29\" (UniqueName: \"kubernetes.io/projected/57e688d9-46c5-47a1-8150-de9e8e024c49-kube-api-access-cvz29\") pod \"57e688d9-46c5-47a1-8150-de9e8e024c49\" (UID: \"57e688d9-46c5-47a1-8150-de9e8e024c49\") "
Oct 30 20:08:35.379430 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.379254    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-workload-socket\") pod \"505b8b25-660e-432f-87b0-f359078e9f2a\" (UID: \"505b8b25-660e-432f-87b0-f359078e9f2a\") "
Oct 30 20:08:35.380127 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.379518    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e688d9-46c5-47a1-8150-de9e8e024c49-istiod-ca-cert" pod "57e688d9-46c5-47a1-8150-de9e8e024c49" (UID: "57e688d9-46c5-47a1-8150-de9e8e024c49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:35.380127 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.379992    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-istio-data" pod "505b8b25-660e-432f-87b0-f359078e9f2a" (UID: "505b8b25-660e-432f-87b0-f359078e9f2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:35.381075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.380888    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-workload-socket" pod "505b8b25-660e-432f-87b0-f359078e9f2a" (UID: "505b8b25-660e-432f-87b0-f359078e9f2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:35.381075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.380919    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e688d9-46c5-47a1-8150-de9e8e024c49-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.381075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.380965    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-credential-socket" pod "505b8b25-660e-432f-87b0-f359078e9f2a" (UID: "505b8b25-660e-432f-87b0-f359078e9f2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:35.381075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.380988    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.381075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.381008    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.381075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.381065    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.381514 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.381083    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.381514 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.381126    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.381514 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.381272    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-credential-socket" pod "57e688d9-46c5-47a1-8150-de9e8e024c49" (UID: "57e688d9-46c5-47a1-8150-de9e8e024c49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:35.383409 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.383381    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-istio-envoy" pod "505b8b25-660e-432f-87b0-f359078e9f2a" (UID: "505b8b25-660e-432f-87b0-f359078e9f2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:35.383524 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.383406    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/505b8b25-660e-432f-87b0-f359078e9f2a-istiod-ca-cert" pod "505b8b25-660e-432f-87b0-f359078e9f2a" (UID: "505b8b25-660e-432f-87b0-f359078e9f2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:35.384987 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.384960    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505b8b25-660e-432f-87b0-f359078e9f2a-kube-api-access-cmwmz" pod "505b8b25-660e-432f-87b0-f359078e9f2a" (UID: "505b8b25-660e-432f-87b0-f359078e9f2a"). InnerVolumeSpecName "kube-api-access-cmwmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:35.385134 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.385007    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505b8b25-660e-432f-87b0-f359078e9f2a-istio-token" pod "505b8b25-660e-432f-87b0-f359078e9f2a" (UID: "505b8b25-660e-432f-87b0-f359078e9f2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:35.385518 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.385491    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-podinfo" pod "57e688d9-46c5-47a1-8150-de9e8e024c49" (UID: "57e688d9-46c5-47a1-8150-de9e8e024c49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:08:35.385767 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.385514    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-token" pod "57e688d9-46c5-47a1-8150-de9e8e024c49" (UID: "57e688d9-46c5-47a1-8150-de9e8e024c49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:35.385856 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.385827    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e688d9-46c5-47a1-8150-de9e8e024c49-kube-api-access-cvz29" pod "57e688d9-46c5-47a1-8150-de9e8e024c49" (UID: "57e688d9-46c5-47a1-8150-de9e8e024c49"). InnerVolumeSpecName "kube-api-access-cvz29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:35.386942 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.386914    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/505b8b25-660e-432f-87b0-f359078e9f2a-istio-podinfo" pod "505b8b25-660e-432f-87b0-f359078e9f2a" (UID: "505b8b25-660e-432f-87b0-f359078e9f2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:08:35.387143 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.387121    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-envoy" pod "57e688d9-46c5-47a1-8150-de9e8e024c49" (UID: "57e688d9-46c5-47a1-8150-de9e8e024c49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:35.481918 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.481885    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505b8b25-660e-432f-87b0-f359078e9f2a-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.481918 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.481912    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.481918 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.481922    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.482190 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.481933    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.482190 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.481942    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e688d9-46c5-47a1-8150-de9e8e024c49-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.482190 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.481951    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505b8b25-660e-432f-87b0-f359078e9f2a-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.482190 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.481959    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.482190 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.481969    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmwmz\" (UniqueName: \"kubernetes.io/projected/505b8b25-660e-432f-87b0-f359078e9f2a-kube-api-access-cmwmz\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.482190 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.481978    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvz29\" (UniqueName: \"kubernetes.io/projected/57e688d9-46c5-47a1-8150-de9e8e024c49-kube-api-access-cvz29\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.482190 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.481988    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505b8b25-660e-432f-87b0-f359078e9f2a-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.482190 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.481996    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505b8b25-660e-432f-87b0-f359078e9f2a-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.482190 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.482004    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e688d9-46c5-47a1-8150-de9e8e024c49-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:35.771194 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.771099    6030 generic.go:356] "Generic (PLEG): container finished" podID="57e688d9-46c5-47a1-8150-de9e8e024c49" containerID="ee2d2aa112d61de791a8e6e250487c0daf1922773f771b8bbda09cd3f7751e92" exitCode=0
Oct 30 20:08:35.771194 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.771170    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"
Oct 30 20:08:35.771434 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.771183    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l" event={"ID":"57e688d9-46c5-47a1-8150-de9e8e024c49","Type":"ContainerDied","Data":"ee2d2aa112d61de791a8e6e250487c0daf1922773f771b8bbda09cd3f7751e92"}
Oct 30 20:08:35.771434 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.771242    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l" event={"ID":"57e688d9-46c5-47a1-8150-de9e8e024c49","Type":"ContainerDied","Data":"212d43f70138f796c317c87ee263c13b9b672d9b91bc48cea7bb0fb1a8febf35"}
Oct 30 20:08:35.771434 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.771270    6030 scope.go:122] "RemoveContainer" containerID="ee2d2aa112d61de791a8e6e250487c0daf1922773f771b8bbda09cd3f7751e92"
Oct 30 20:08:35.773127 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.773105    6030 generic.go:356] "Generic (PLEG): container finished" podID="505b8b25-660e-432f-87b0-f359078e9f2a" containerID="e715322eac72ff1c96c639b90f4fb91fe32c3e0a14647cd05480cdcf30e7ab1a" exitCode=0
Oct 30 20:08:35.773446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.773165    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9" event={"ID":"505b8b25-660e-432f-87b0-f359078e9f2a","Type":"ContainerDied","Data":"e715322eac72ff1c96c639b90f4fb91fe32c3e0a14647cd05480cdcf30e7ab1a"}
Oct 30 20:08:35.773581 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.773479    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9" event={"ID":"505b8b25-660e-432f-87b0-f359078e9f2a","Type":"ContainerDied","Data":"c8120bd38425fdeea9f04953e1e7d726e17a77b0882ea4c8becd993d2c4fdb22"}
Oct 30 20:08:35.773581 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.773219    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"
Oct 30 20:08:35.783503 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.783481    6030 scope.go:122] "RemoveContainer" containerID="ee2d2aa112d61de791a8e6e250487c0daf1922773f771b8bbda09cd3f7751e92"
Oct 30 20:08:35.783929 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:08:35.783897    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee2d2aa112d61de791a8e6e250487c0daf1922773f771b8bbda09cd3f7751e92\": container with ID starting with ee2d2aa112d61de791a8e6e250487c0daf1922773f771b8bbda09cd3f7751e92 not found: ID does not exist" containerID="ee2d2aa112d61de791a8e6e250487c0daf1922773f771b8bbda09cd3f7751e92"
Oct 30 20:08:35.784012 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.783931    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2d2aa112d61de791a8e6e250487c0daf1922773f771b8bbda09cd3f7751e92"} err="failed to get container status \"ee2d2aa112d61de791a8e6e250487c0daf1922773f771b8bbda09cd3f7751e92\": rpc error: code = NotFound desc = could not find container \"ee2d2aa112d61de791a8e6e250487c0daf1922773f771b8bbda09cd3f7751e92\": container with ID starting with ee2d2aa112d61de791a8e6e250487c0daf1922773f771b8bbda09cd3f7751e92 not found: ID does not exist"
Oct 30 20:08:35.784012 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.783954    6030 scope.go:122] "RemoveContainer" containerID="e715322eac72ff1c96c639b90f4fb91fe32c3e0a14647cd05480cdcf30e7ab1a"
Oct 30 20:08:35.795502 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.795471    6030 scope.go:122] "RemoveContainer" containerID="e715322eac72ff1c96c639b90f4fb91fe32c3e0a14647cd05480cdcf30e7ab1a"
Oct 30 20:08:35.795809 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:08:35.795782    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e715322eac72ff1c96c639b90f4fb91fe32c3e0a14647cd05480cdcf30e7ab1a\": container with ID starting with e715322eac72ff1c96c639b90f4fb91fe32c3e0a14647cd05480cdcf30e7ab1a not found: ID does not exist" containerID="e715322eac72ff1c96c639b90f4fb91fe32c3e0a14647cd05480cdcf30e7ab1a"
Oct 30 20:08:35.795915 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.795818    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e715322eac72ff1c96c639b90f4fb91fe32c3e0a14647cd05480cdcf30e7ab1a"} err="failed to get container status \"e715322eac72ff1c96c639b90f4fb91fe32c3e0a14647cd05480cdcf30e7ab1a\": rpc error: code = NotFound desc = could not find container \"e715322eac72ff1c96c639b90f4fb91fe32c3e0a14647cd05480cdcf30e7ab1a\": container with ID starting with e715322eac72ff1c96c639b90f4fb91fe32c3e0a14647cd05480cdcf30e7ab1a not found: ID does not exist"
Oct 30 20:08:35.809096 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.808996    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"]
Oct 30 20:08:35.812634 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.812606    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-sfv6l"]
Oct 30 20:08:35.830536 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.830496    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"]
Oct 30 20:08:35.832925 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:35.832901    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-9jqb9"]
Oct 30 20:08:36.714369 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.714323    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"]
Oct 30 20:08:36.715364 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.715338    6030 cpu_manager.go:414] "containerMap: removing container" podUID="505b8b25-660e-432f-87b0-f359078e9f2a" containerName="istio-proxy"
Oct 30 20:08:36.715364 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.715364    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="505b8b25-660e-432f-87b0-f359078e9f2a" containerName="istio-proxy"
Oct 30 20:08:36.715575 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.715395    6030 cpu_manager.go:414] "containerMap: removing container" podUID="57e688d9-46c5-47a1-8150-de9e8e024c49" containerName="istio-proxy"
Oct 30 20:08:36.715575 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.715402    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e688d9-46c5-47a1-8150-de9e8e024c49" containerName="istio-proxy"
Oct 30 20:08:36.715575 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.715566    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="505b8b25-660e-432f-87b0-f359078e9f2a" containerName="istio-proxy"
Oct 30 20:08:36.715736 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.715583    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e688d9-46c5-47a1-8150-de9e8e024c49" containerName="istio-proxy"
Oct 30 20:08:36.721111 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.721089    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:08:36.728581 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.728559    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"routing-overlay\""
Oct 30 20:08:36.728646 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.728632    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"payload-processing-external-model-plugins-e2e-extmod-a-222480\""
Oct 30 20:08:36.728710 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.728696    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"kube-root-ca.crt\""
Oct 30 20:08:36.728768 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.728741    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"openshift-service-ca.crt\""
Oct 30 20:08:36.728923 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.728903    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"agc-cred-76d75295798f\""
Oct 30 20:08:36.729276 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.729264    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"payload-processing-external-model-e2e--8e4491f3-dockercfg-q7pzc\""
Oct 30 20:08:36.736009 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.735979    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"]
Oct 30 20:08:36.899762 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.899726    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b00abc9d-1729-4416-ab87-55e3a0f9ad64-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc\" (UID: \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:08:36.900064 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.900024    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b00abc9d-1729-4416-ab87-55e3a0f9ad64-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc\" (UID: \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:08:36.900262 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.900241    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b00abc9d-1729-4416-ab87-55e3a0f9ad64-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc\" (UID: \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:08:36.900462 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:36.900446    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2cbz\" (UniqueName: \"kubernetes.io/projected/b00abc9d-1729-4416-ab87-55e3a0f9ad64-kube-api-access-p2cbz\") pod \"payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc\" (UID: \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:08:37.001627 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:37.001547    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b00abc9d-1729-4416-ab87-55e3a0f9ad64-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc\" (UID: \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:08:37.001786 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:37.001622    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b00abc9d-1729-4416-ab87-55e3a0f9ad64-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc\" (UID: \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:08:37.001786 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:37.001738    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2cbz\" (UniqueName: \"kubernetes.io/projected/b00abc9d-1729-4416-ab87-55e3a0f9ad64-kube-api-access-p2cbz\") pod \"payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc\" (UID: \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:08:37.001921 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:37.001902    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b00abc9d-1729-4416-ab87-55e3a0f9ad64-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc\" (UID: \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:08:37.002355 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:37.002334    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b00abc9d-1729-4416-ab87-55e3a0f9ad64-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc\" (UID: \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:08:37.002530 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:37.002503    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b00abc9d-1729-4416-ab87-55e3a0f9ad64-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc\" (UID: \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:08:37.004536 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:37.004512    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b00abc9d-1729-4416-ab87-55e3a0f9ad64-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc\" (UID: \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:08:37.022674 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:37.022649    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2cbz\" (UniqueName: \"kubernetes.io/projected/b00abc9d-1729-4416-ab87-55e3a0f9ad64-kube-api-access-p2cbz\") pod \"payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc\" (UID: \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:08:37.036586 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:37.036559    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:08:37.203650 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:37.203618    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="505b8b25-660e-432f-87b0-f359078e9f2a" path="/var/lib/kubelet/pods/505b8b25-660e-432f-87b0-f359078e9f2a/volumes"
Oct 30 20:08:37.204248 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:37.204233    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e688d9-46c5-47a1-8150-de9e8e024c49" path="/var/lib/kubelet/pods/57e688d9-46c5-47a1-8150-de9e8e024c49/volumes"
Oct 30 20:08:37.341271 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:37.341249    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"]
Oct 30 20:08:37.342152 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:08:37.342122    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00abc9d_1729_4416_ab87_55e3a0f9ad64.slice/crio-d74c6ec1bd7d94aabf6633efc589587a41676c583843c2c3c062d45801d95ef2 WatchSource:0}: Error finding container d74c6ec1bd7d94aabf6633efc589587a41676c583843c2c3c062d45801d95ef2: Status 404 returned error can't find the container with id d74c6ec1bd7d94aabf6633efc589587a41676c583843c2c3c062d45801d95ef2
Oct 30 20:08:37.924979 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:37.924942    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc" event={"ID":"b00abc9d-1729-4416-ab87-55e3a0f9ad64","Type":"ContainerStarted","Data":"34f91b6e85d663a7aa36af4f0e327432a2e3a9ba3bc4a1f3adc39ea72481606b"}
Oct 30 20:08:37.925700 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:37.925011    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:08:37.925700 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:37.925034    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc" event={"ID":"b00abc9d-1729-4416-ab87-55e3a0f9ad64","Type":"ContainerStarted","Data":"d74c6ec1bd7d94aabf6633efc589587a41676c583843c2c3c062d45801d95ef2"}
Oct 30 20:08:37.957401 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:37.957354    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc" podStartSLOduration=1.957339959 podStartE2EDuration="1.957339959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:37.953118492 +0000 UTC m=+2196.868725349" watchObservedRunningTime="2026-10-30 20:08:37.957339959 +0000 UTC m=+2196.872946815"
Oct 30 20:08:44.397084 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:44.393745    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"]
Oct 30 20:08:48.936295 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:48.936261    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:08:49.394913 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:49.394874    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n" podUID="4a3cf286-3aa5-47a2-b87a-a698306d041d" containerName="payload-processing" containerID="cri-o://359bc5098d81695eebb5c5a4a6fea9a60538eb81d8bb38dc83523a86d3c8a7cd" gracePeriod=25
Oct 30 20:08:49.411953 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:08:49.411931    6030 logging.go:55] [core] [Channel #4178 SubChannel #4179]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.178:50052", ServerName: "10.133.0.178:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.178:50052: connect: connection refused"
Oct 30 20:08:49.680105 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:49.680066    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:08:49.716203 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:49.716177    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3cf286-3aa5-47a2-b87a-a698306d041d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a3cf286-3aa5-47a2-b87a-a698306d041d-provider-credentials\") pod \"4a3cf286-3aa5-47a2-b87a-a698306d041d\" (UID: \"4a3cf286-3aa5-47a2-b87a-a698306d041d\") "
Oct 30 20:08:49.716355 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:49.716213    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3cf286-3aa5-47a2-b87a-a698306d041d-kube-api-access-7gftt\" (UniqueName: \"kubernetes.io/projected/4a3cf286-3aa5-47a2-b87a-a698306d041d-kube-api-access-7gftt\") pod \"4a3cf286-3aa5-47a2-b87a-a698306d041d\" (UID: \"4a3cf286-3aa5-47a2-b87a-a698306d041d\") "
Oct 30 20:08:49.716355 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:49.716323    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a3cf286-3aa5-47a2-b87a-a698306d041d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a3cf286-3aa5-47a2-b87a-a698306d041d-routing-overlay-volume\") pod \"4a3cf286-3aa5-47a2-b87a-a698306d041d\" (UID: \"4a3cf286-3aa5-47a2-b87a-a698306d041d\") "
Oct 30 20:08:49.716470 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:49.716399    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a3cf286-3aa5-47a2-b87a-a698306d041d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3cf286-3aa5-47a2-b87a-a698306d041d-plugins-config-volume\") pod \"4a3cf286-3aa5-47a2-b87a-a698306d041d\" (UID: \"4a3cf286-3aa5-47a2-b87a-a698306d041d\") "
Oct 30 20:08:49.716755 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:49.716723    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3cf286-3aa5-47a2-b87a-a698306d041d-routing-overlay-volume" pod "4a3cf286-3aa5-47a2-b87a-a698306d041d" (UID: "4a3cf286-3aa5-47a2-b87a-a698306d041d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:49.716857 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:49.716812    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3cf286-3aa5-47a2-b87a-a698306d041d-plugins-config-volume" pod "4a3cf286-3aa5-47a2-b87a-a698306d041d" (UID: "4a3cf286-3aa5-47a2-b87a-a698306d041d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:49.717018 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:49.717000    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a3cf286-3aa5-47a2-b87a-a698306d041d-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:49.717114 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:49.717023    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3cf286-3aa5-47a2-b87a-a698306d041d-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:49.719359 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:49.719333    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3cf286-3aa5-47a2-b87a-a698306d041d-provider-credentials" pod "4a3cf286-3aa5-47a2-b87a-a698306d041d" (UID: "4a3cf286-3aa5-47a2-b87a-a698306d041d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:49.719452 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:49.719357    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3cf286-3aa5-47a2-b87a-a698306d041d-kube-api-access-7gftt" pod "4a3cf286-3aa5-47a2-b87a-a698306d041d" (UID: "4a3cf286-3aa5-47a2-b87a-a698306d041d"). InnerVolumeSpecName "kube-api-access-7gftt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:49.861068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:49.830221    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a3cf286-3aa5-47a2-b87a-a698306d041d-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:49.864227 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:49.861313    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gftt\" (UniqueName: \"kubernetes.io/projected/4a3cf286-3aa5-47a2-b87a-a698306d041d-kube-api-access-7gftt\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:50.114370 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:50.114278    6030 generic.go:356] "Generic (PLEG): container finished" podID="4a3cf286-3aa5-47a2-b87a-a698306d041d" containerID="359bc5098d81695eebb5c5a4a6fea9a60538eb81d8bb38dc83523a86d3c8a7cd" exitCode=0
Oct 30 20:08:50.114835 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:50.114610    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n" event={"ID":"4a3cf286-3aa5-47a2-b87a-a698306d041d","Type":"ContainerDied","Data":"359bc5098d81695eebb5c5a4a6fea9a60538eb81d8bb38dc83523a86d3c8a7cd"}
Oct 30 20:08:50.114835 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:50.114709    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n" event={"ID":"4a3cf286-3aa5-47a2-b87a-a698306d041d","Type":"ContainerDied","Data":"1ee7d1089c24141c5ff0238138e4aa47467720e9dfe1bc09b8dc7cb864a16b6e"}
Oct 30 20:08:50.114835 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:50.114747    6030 scope.go:122] "RemoveContainer" containerID="359bc5098d81695eebb5c5a4a6fea9a60538eb81d8bb38dc83523a86d3c8a7cd"
Oct 30 20:08:50.114982 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:50.114970    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"
Oct 30 20:08:50.133011 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:50.132943    6030 scope.go:122] "RemoveContainer" containerID="359bc5098d81695eebb5c5a4a6fea9a60538eb81d8bb38dc83523a86d3c8a7cd"
Oct 30 20:08:50.133477 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:08:50.133438    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359bc5098d81695eebb5c5a4a6fea9a60538eb81d8bb38dc83523a86d3c8a7cd\": container with ID starting with 359bc5098d81695eebb5c5a4a6fea9a60538eb81d8bb38dc83523a86d3c8a7cd not found: ID does not exist" containerID="359bc5098d81695eebb5c5a4a6fea9a60538eb81d8bb38dc83523a86d3c8a7cd"
Oct 30 20:08:50.133607 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:50.133480    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359bc5098d81695eebb5c5a4a6fea9a60538eb81d8bb38dc83523a86d3c8a7cd"} err="failed to get container status \"359bc5098d81695eebb5c5a4a6fea9a60538eb81d8bb38dc83523a86d3c8a7cd\": rpc error: code = NotFound desc = could not find container \"359bc5098d81695eebb5c5a4a6fea9a60538eb81d8bb38dc83523a86d3c8a7cd\": container with ID starting with 359bc5098d81695eebb5c5a4a6fea9a60538eb81d8bb38dc83523a86d3c8a7cd not found: ID does not exist"
Oct 30 20:08:50.161532 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:50.161504    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"]
Oct 30 20:08:50.170738 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:50.170708    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n"]
Oct 30 20:08:50.412403 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:50.412366    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-9f67p546n" podUID="4a3cf286-3aa5-47a2-b87a-a698306d041d" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.178:50052\" within 1s: context deadline exceeded"
Oct 30 20:08:51.204100 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:08:51.204071    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a3cf286-3aa5-47a2-b87a-a698306d041d" path="/var/lib/kubelet/pods/4a3cf286-3aa5-47a2-b87a-a698306d041d/volumes"
Oct 30 20:09:04.289708 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:09:04.289664    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:09:07.058915 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.058880    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"]
Oct 30 20:09:07.059567 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.059552    6030 cpu_manager.go:414] "containerMap: removing container" podUID="4a3cf286-3aa5-47a2-b87a-a698306d041d" containerName="payload-processing"
Oct 30 20:09:07.059567 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.059568    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a3cf286-3aa5-47a2-b87a-a698306d041d" containerName="payload-processing"
Oct 30 20:09:07.059712 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.059702    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a3cf286-3aa5-47a2-b87a-a698306d041d" containerName="payload-processing"
Oct 30 20:09:07.064934 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.064913    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:07.073830 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.073807    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 20:09:07.076366 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.076341    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 20:09:07.076522 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.076350    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 20:09:07.076522 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.076413    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 20:09:07.093860 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.093841    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"]
Oct 30 20:09:07.143337 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.143309    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-mqtgc\""
Oct 30 20:09:07.143801 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.143770    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9d45aadaaa09\""
Oct 30 20:09:07.144026 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.143995    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-08c2badb5904\""
Oct 30 20:09:07.169520 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.169494    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c041341-e446-4c3d-b844-85e6a8091374-plugins-config-volume\") pod \"payload-processing-external-model-5ddc6fbc79-8zwfn\" (UID: \"6c041341-e446-4c3d-b844-85e6a8091374\") " pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:07.169660 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.169534    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c041341-e446-4c3d-b844-85e6a8091374-provider-credentials\") pod \"payload-processing-external-model-5ddc6fbc79-8zwfn\" (UID: \"6c041341-e446-4c3d-b844-85e6a8091374\") " pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:07.169727 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.169658    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfg6b\" (UniqueName: \"kubernetes.io/projected/6c041341-e446-4c3d-b844-85e6a8091374-kube-api-access-cfg6b\") pod \"payload-processing-external-model-5ddc6fbc79-8zwfn\" (UID: \"6c041341-e446-4c3d-b844-85e6a8091374\") " pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:07.169727 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.169709    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c041341-e446-4c3d-b844-85e6a8091374-routing-overlay-volume\") pod \"payload-processing-external-model-5ddc6fbc79-8zwfn\" (UID: \"6c041341-e446-4c3d-b844-85e6a8091374\") " pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:07.271672 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.271628    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfg6b\" (UniqueName: \"kubernetes.io/projected/6c041341-e446-4c3d-b844-85e6a8091374-kube-api-access-cfg6b\") pod \"payload-processing-external-model-5ddc6fbc79-8zwfn\" (UID: \"6c041341-e446-4c3d-b844-85e6a8091374\") " pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:07.271846 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.271722    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c041341-e446-4c3d-b844-85e6a8091374-routing-overlay-volume\") pod \"payload-processing-external-model-5ddc6fbc79-8zwfn\" (UID: \"6c041341-e446-4c3d-b844-85e6a8091374\") " pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:07.271959 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.271942    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c041341-e446-4c3d-b844-85e6a8091374-plugins-config-volume\") pod \"payload-processing-external-model-5ddc6fbc79-8zwfn\" (UID: \"6c041341-e446-4c3d-b844-85e6a8091374\") " pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:07.272007 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.271991    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c041341-e446-4c3d-b844-85e6a8091374-provider-credentials\") pod \"payload-processing-external-model-5ddc6fbc79-8zwfn\" (UID: \"6c041341-e446-4c3d-b844-85e6a8091374\") " pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:07.272593 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.272558    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c041341-e446-4c3d-b844-85e6a8091374-routing-overlay-volume\") pod \"payload-processing-external-model-5ddc6fbc79-8zwfn\" (UID: \"6c041341-e446-4c3d-b844-85e6a8091374\") " pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:07.272745 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.272610    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c041341-e446-4c3d-b844-85e6a8091374-plugins-config-volume\") pod \"payload-processing-external-model-5ddc6fbc79-8zwfn\" (UID: \"6c041341-e446-4c3d-b844-85e6a8091374\") " pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:07.274876 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.274852    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c041341-e446-4c3d-b844-85e6a8091374-provider-credentials\") pod \"payload-processing-external-model-5ddc6fbc79-8zwfn\" (UID: \"6c041341-e446-4c3d-b844-85e6a8091374\") " pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:07.291112 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.291038    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfg6b\" (UniqueName: \"kubernetes.io/projected/6c041341-e446-4c3d-b844-85e6a8091374-kube-api-access-cfg6b\") pod \"payload-processing-external-model-5ddc6fbc79-8zwfn\" (UID: \"6c041341-e446-4c3d-b844-85e6a8091374\") " pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:07.381424 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.381401    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:07.678036 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:09:07.677301    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c041341_e446_4c3d_b844_85e6a8091374.slice/crio-a5387774f80777435efeb10f65d6bf85a9b9bd8882978b25e761d994e8d30b08 WatchSource:0}: Error finding container a5387774f80777435efeb10f65d6bf85a9b9bd8882978b25e761d994e8d30b08: Status 404 returned error can't find the container with id a5387774f80777435efeb10f65d6bf85a9b9bd8882978b25e761d994e8d30b08
Oct 30 20:09:07.679140 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:07.678986    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"]
Oct 30 20:09:08.296068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:08.280717    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn" event={"ID":"6c041341-e446-4c3d-b844-85e6a8091374","Type":"ContainerStarted","Data":"d3b200c60b1f46f2d7c271a885a0f3b8988da4f57cad1baa9e82752ae39b3581"}
Oct 30 20:09:08.296068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:08.280826    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn" event={"ID":"6c041341-e446-4c3d-b844-85e6a8091374","Type":"ContainerStarted","Data":"a5387774f80777435efeb10f65d6bf85a9b9bd8882978b25e761d994e8d30b08"}
Oct 30 20:09:08.296068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:08.281976    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:08.343069 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:08.340638    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn" podStartSLOduration=1.34061778 podStartE2EDuration="1.34061778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:08.320937912 +0000 UTC m=+2227.236544769" watchObservedRunningTime="2026-10-30 20:09:08.34061778 +0000 UTC m=+2227.256224637"
Oct 30 20:09:20.298135 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:20.298014    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:23.363245 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:23.363199    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"]
Oct 30 20:09:28.364010 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:28.363961    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn" podUID="6c041341-e446-4c3d-b844-85e6a8091374" containerName="payload-processing" containerID="cri-o://d3b200c60b1f46f2d7c271a885a0f3b8988da4f57cad1baa9e82752ae39b3581" gracePeriod=25
Oct 30 20:09:28.727601 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:28.727567    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:28.754114 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:28.754090    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c041341-e446-4c3d-b844-85e6a8091374-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c041341-e446-4c3d-b844-85e6a8091374-provider-credentials\") pod \"6c041341-e446-4c3d-b844-85e6a8091374\" (UID: \"6c041341-e446-4c3d-b844-85e6a8091374\") "
Oct 30 20:09:28.754228 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:28.754201    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c041341-e446-4c3d-b844-85e6a8091374-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c041341-e446-4c3d-b844-85e6a8091374-routing-overlay-volume\") pod \"6c041341-e446-4c3d-b844-85e6a8091374\" (UID: \"6c041341-e446-4c3d-b844-85e6a8091374\") "
Oct 30 20:09:28.754271 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:28.754241    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c041341-e446-4c3d-b844-85e6a8091374-kube-api-access-cfg6b\" (UniqueName: \"kubernetes.io/projected/6c041341-e446-4c3d-b844-85e6a8091374-kube-api-access-cfg6b\") pod \"6c041341-e446-4c3d-b844-85e6a8091374\" (UID: \"6c041341-e446-4c3d-b844-85e6a8091374\") "
Oct 30 20:09:28.754404 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:28.754386    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c041341-e446-4c3d-b844-85e6a8091374-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c041341-e446-4c3d-b844-85e6a8091374-plugins-config-volume\") pod \"6c041341-e446-4c3d-b844-85e6a8091374\" (UID: \"6c041341-e446-4c3d-b844-85e6a8091374\") "
Oct 30 20:09:28.754668 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:28.754639    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c041341-e446-4c3d-b844-85e6a8091374-routing-overlay-volume" pod "6c041341-e446-4c3d-b844-85e6a8091374" (UID: "6c041341-e446-4c3d-b844-85e6a8091374"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:09:28.754782 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:28.754757    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c041341-e446-4c3d-b844-85e6a8091374-plugins-config-volume" pod "6c041341-e446-4c3d-b844-85e6a8091374" (UID: "6c041341-e446-4c3d-b844-85e6a8091374"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:09:28.754953 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:28.754937    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c041341-e446-4c3d-b844-85e6a8091374-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:28.754998 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:28.754956    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c041341-e446-4c3d-b844-85e6a8091374-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:28.756294 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:28.756267    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c041341-e446-4c3d-b844-85e6a8091374-provider-credentials" pod "6c041341-e446-4c3d-b844-85e6a8091374" (UID: "6c041341-e446-4c3d-b844-85e6a8091374"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:28.756590 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:28.756569    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c041341-e446-4c3d-b844-85e6a8091374-kube-api-access-cfg6b" pod "6c041341-e446-4c3d-b844-85e6a8091374" (UID: "6c041341-e446-4c3d-b844-85e6a8091374"). InnerVolumeSpecName "kube-api-access-cfg6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:28.856578 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:28.856543    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c041341-e446-4c3d-b844-85e6a8091374-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:28.856578 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:28.856574    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfg6b\" (UniqueName: \"kubernetes.io/projected/6c041341-e446-4c3d-b844-85e6a8091374-kube-api-access-cfg6b\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:29.451004 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:29.450973    6030 generic.go:356] "Generic (PLEG): container finished" podID="6c041341-e446-4c3d-b844-85e6a8091374" containerID="d3b200c60b1f46f2d7c271a885a0f3b8988da4f57cad1baa9e82752ae39b3581" exitCode=0
Oct 30 20:09:29.451490 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:29.451012    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn" event={"ID":"6c041341-e446-4c3d-b844-85e6a8091374","Type":"ContainerDied","Data":"d3b200c60b1f46f2d7c271a885a0f3b8988da4f57cad1baa9e82752ae39b3581"}
Oct 30 20:09:29.451490 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:29.451017    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"
Oct 30 20:09:29.451490 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:29.451075    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn" event={"ID":"6c041341-e446-4c3d-b844-85e6a8091374","Type":"ContainerDied","Data":"a5387774f80777435efeb10f65d6bf85a9b9bd8882978b25e761d994e8d30b08"}
Oct 30 20:09:29.451490 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:29.451099    6030 scope.go:122] "RemoveContainer" containerID="d3b200c60b1f46f2d7c271a885a0f3b8988da4f57cad1baa9e82752ae39b3581"
Oct 30 20:09:29.461694 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:29.461676    6030 scope.go:122] "RemoveContainer" containerID="d3b200c60b1f46f2d7c271a885a0f3b8988da4f57cad1baa9e82752ae39b3581"
Oct 30 20:09:29.461992 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:09:29.461970    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b200c60b1f46f2d7c271a885a0f3b8988da4f57cad1baa9e82752ae39b3581\": container with ID starting with d3b200c60b1f46f2d7c271a885a0f3b8988da4f57cad1baa9e82752ae39b3581 not found: ID does not exist" containerID="d3b200c60b1f46f2d7c271a885a0f3b8988da4f57cad1baa9e82752ae39b3581"
Oct 30 20:09:29.462064 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:29.462000    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b200c60b1f46f2d7c271a885a0f3b8988da4f57cad1baa9e82752ae39b3581"} err="failed to get container status \"d3b200c60b1f46f2d7c271a885a0f3b8988da4f57cad1baa9e82752ae39b3581\": rpc error: code = NotFound desc = could not find container \"d3b200c60b1f46f2d7c271a885a0f3b8988da4f57cad1baa9e82752ae39b3581\": container with ID starting with d3b200c60b1f46f2d7c271a885a0f3b8988da4f57cad1baa9e82752ae39b3581 not found: ID does not exist"
Oct 30 20:09:29.480280 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:29.480214    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"]
Oct 30 20:09:29.488166 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:29.488137    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ddc6fbc79-8zwfn"]
Oct 30 20:09:31.232293 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:31.232257    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c041341-e446-4c3d-b844-85e6a8091374" path="/var/lib/kubelet/pods/6c041341-e446-4c3d-b844-85e6a8091374/volumes"
Oct 30 20:09:34.292870 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:09:34.292835    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:09:40.208484 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.208444    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"]
Oct 30 20:09:40.209616 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.209594    6030 cpu_manager.go:414] "containerMap: removing container" podUID="6c041341-e446-4c3d-b844-85e6a8091374" containerName="payload-processing"
Oct 30 20:09:40.209697 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.209619    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c041341-e446-4c3d-b844-85e6a8091374" containerName="payload-processing"
Oct 30 20:09:40.209857 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.209844    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c041341-e446-4c3d-b844-85e6a8091374" containerName="payload-processing"
Oct 30 20:09:40.217590 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.217563    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"]
Oct 30 20:09:40.217754 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.217631    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:09:40.226473 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.226446    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"openshift-service-ca.crt\""
Oct 30 20:09:40.226632 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.226545    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"kube-root-ca.crt\""
Oct 30 20:09:40.226632 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.226575    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"payload-processing-external-model-e2e--d955aedb-dockercfg-8lgzf\""
Oct 30 20:09:40.226632 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.226595    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c9bce8\""
Oct 30 20:09:40.226632 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.226454    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"routing-overlay\""
Oct 30 20:09:40.226632 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.226627    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"agc-cred-e6b77e70a3b3\""
Oct 30 20:09:40.361160 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.353951    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4a22544-17f5-4072-b5cb-876ed3f2f80b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj\" (UID: \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:09:40.361160 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.354082    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s4xf\" (UniqueName: \"kubernetes.io/projected/b4a22544-17f5-4072-b5cb-876ed3f2f80b-kube-api-access-2s4xf\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj\" (UID: \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:09:40.361160 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.354168    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a22544-17f5-4072-b5cb-876ed3f2f80b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj\" (UID: \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:09:40.361160 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.354397    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4a22544-17f5-4072-b5cb-876ed3f2f80b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj\" (UID: \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:09:40.456298 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.456262    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4a22544-17f5-4072-b5cb-876ed3f2f80b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj\" (UID: \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:09:40.456488 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.456321    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s4xf\" (UniqueName: \"kubernetes.io/projected/b4a22544-17f5-4072-b5cb-876ed3f2f80b-kube-api-access-2s4xf\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj\" (UID: \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:09:40.456488 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.456374    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a22544-17f5-4072-b5cb-876ed3f2f80b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj\" (UID: \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:09:40.456601 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.456523    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4a22544-17f5-4072-b5cb-876ed3f2f80b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj\" (UID: \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:09:40.457457 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.457427    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4a22544-17f5-4072-b5cb-876ed3f2f80b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj\" (UID: \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:09:40.457457 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.457461    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a22544-17f5-4072-b5cb-876ed3f2f80b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj\" (UID: \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:09:40.459595 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.459545    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4a22544-17f5-4072-b5cb-876ed3f2f80b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj\" (UID: \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:09:40.488031 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.484863    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s4xf\" (UniqueName: \"kubernetes.io/projected/b4a22544-17f5-4072-b5cb-876ed3f2f80b-kube-api-access-2s4xf\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj\" (UID: \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:09:40.534914 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.534880    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:09:40.821468 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:40.821435    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"]
Oct 30 20:09:40.822019 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:09:40.821992    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a22544_17f5_4072_b5cb_876ed3f2f80b.slice/crio-86ac4e81b936bfa7db21892beaa8ec0394b699b676a7133ed306098f4e6778bd WatchSource:0}: Error finding container 86ac4e81b936bfa7db21892beaa8ec0394b699b676a7133ed306098f4e6778bd: Status 404 returned error can't find the container with id 86ac4e81b936bfa7db21892beaa8ec0394b699b676a7133ed306098f4e6778bd
Oct 30 20:09:41.590663 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:41.590627    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj" event={"ID":"b4a22544-17f5-4072-b5cb-876ed3f2f80b","Type":"ContainerStarted","Data":"a7959cf952a8448ca154fa795f13ff0b3f2a99dd145186e21cc60bfcb3077296"}
Oct 30 20:09:41.591092 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:41.590679    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj" event={"ID":"b4a22544-17f5-4072-b5cb-876ed3f2f80b","Type":"ContainerStarted","Data":"86ac4e81b936bfa7db21892beaa8ec0394b699b676a7133ed306098f4e6778bd"}
Oct 30 20:09:41.591092 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:41.590760    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:09:41.617471 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:41.617422    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj" podStartSLOduration=1.6174097280000002 podStartE2EDuration="1.617409728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:41.614234809 +0000 UTC m=+2260.529841688" watchObservedRunningTime="2026-10-30 20:09:41.617409728 +0000 UTC m=+2260.533016584"
Oct 30 20:09:51.112036 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.111973    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"]
Oct 30 20:09:51.116934 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.116910    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.132960 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.132937    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"]
Oct 30 20:09:51.134920 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.134894    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.135061 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.134944    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.135061 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.134964    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.135061 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.134999    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.135234 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.135214    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbfzj\" (UniqueName: \"kubernetes.io/projected/a57ba962-389a-4798-a788-cb0dedfa0f5c-kube-api-access-cbfzj\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.135293 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.135250    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.135293 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.135286    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.135421 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.135393    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a57ba962-389a-4798-a788-cb0dedfa0f5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.135512 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.135497    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.174990 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.174955    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"]
Oct 30 20:09:51.179803 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.179782    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.196025 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.195992    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"]
Oct 30 20:09:51.236901 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.236867    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flfd8\" (UniqueName: \"kubernetes.io/projected/db939b9a-f127-428e-bc8f-c137346b089c-kube-api-access-flfd8\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.237082 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.236906    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.237082 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.236939    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db939b9a-f127-428e-bc8f-c137346b089c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.237204 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.237128    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.237300 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.237263    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db939b9a-f127-428e-bc8f-c137346b089c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.237378 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.237325    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.237378 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.237351    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.237498 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.237388    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.237498 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.237415    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db939b9a-f127-428e-bc8f-c137346b089c-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.237498 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.237476    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db939b9a-f127-428e-bc8f-c137346b089c-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.237666 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.237519    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db939b9a-f127-428e-bc8f-c137346b089c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.237666 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.237580    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.237666 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.237628    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.237833 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.237808    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbfzj\" (UniqueName: \"kubernetes.io/projected/a57ba962-389a-4798-a788-cb0dedfa0f5c-kube-api-access-cbfzj\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.237951 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.237929    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.238034 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.238005    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.238132 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.238084    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db939b9a-f127-428e-bc8f-c137346b089c-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.238191 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.238122    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.238191 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.238168    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db939b9a-f127-428e-bc8f-c137346b089c-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.238306 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.238231    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db939b9a-f127-428e-bc8f-c137346b089c-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.238306 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.238270    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a57ba962-389a-4798-a788-cb0dedfa0f5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.238433 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.238303    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.239007 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.238982    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a57ba962-389a-4798-a788-cb0dedfa0f5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.239921 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.239897    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.240036 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.240010    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.261341 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.261307    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.262757 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.262734    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbfzj\" (UniqueName: \"kubernetes.io/projected/a57ba962-389a-4798-a788-cb0dedfa0f5c-kube-api-access-cbfzj\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mhqnr\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.339647 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.339593    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flfd8\" (UniqueName: \"kubernetes.io/projected/db939b9a-f127-428e-bc8f-c137346b089c-kube-api-access-flfd8\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.339647 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.339655    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db939b9a-f127-428e-bc8f-c137346b089c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.339856 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.339704    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db939b9a-f127-428e-bc8f-c137346b089c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.339856 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.339834    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db939b9a-f127-428e-bc8f-c137346b089c-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.339974 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.339923    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db939b9a-f127-428e-bc8f-c137346b089c-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.340034 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.339982    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db939b9a-f127-428e-bc8f-c137346b089c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.340173 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.340152    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db939b9a-f127-428e-bc8f-c137346b089c-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.340256 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.340214    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db939b9a-f127-428e-bc8f-c137346b089c-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.340314 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.340277    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db939b9a-f127-428e-bc8f-c137346b089c-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.340378 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.340323    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db939b9a-f127-428e-bc8f-c137346b089c-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.340440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.340421    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db939b9a-f127-428e-bc8f-c137346b089c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.340541 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.340520    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db939b9a-f127-428e-bc8f-c137346b089c-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.343106 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.340985    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db939b9a-f127-428e-bc8f-c137346b089c-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.343106 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.340999    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db939b9a-f127-428e-bc8f-c137346b089c-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.343106 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.342402    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db939b9a-f127-428e-bc8f-c137346b089c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.343106 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.342696    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db939b9a-f127-428e-bc8f-c137346b089c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.353575 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.353542    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flfd8\" (UniqueName: \"kubernetes.io/projected/db939b9a-f127-428e-bc8f-c137346b089c-kube-api-access-flfd8\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.356993 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.356972    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db939b9a-f127-428e-bc8f-c137346b089c-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-x6hbg\" (UID: \"db939b9a-f127-428e-bc8f-c137346b089c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.434765 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.434735    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:51.514614 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.514583    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:51.730877 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.730835    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"]
Oct 30 20:09:51.731122 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:09:51.731097    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57ba962_389a_4798_a788_cb0dedfa0f5c.slice/crio-c3e37570b11c4d903de1e9e10fd902e5805d2f4a23503cee95a3112a9b7173cd WatchSource:0}: Error finding container c3e37570b11c4d903de1e9e10fd902e5805d2f4a23503cee95a3112a9b7173cd: Status 404 returned error can't find the container with id c3e37570b11c4d903de1e9e10fd902e5805d2f4a23503cee95a3112a9b7173cd
Oct 30 20:09:51.734009 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.733975    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:09:51.734119 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.734063    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:09:51.734119 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.734107    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:09:51.818171 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:09:51.818143    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb939b9a_f127_428e_bc8f_c137346b089c.slice/crio-bff7403d2443a6acc9960ae2703cd95bf675a6f3ff711e0d626f058b273c6be5 WatchSource:0}: Error finding container bff7403d2443a6acc9960ae2703cd95bf675a6f3ff711e0d626f058b273c6be5: Status 404 returned error can't find the container with id bff7403d2443a6acc9960ae2703cd95bf675a6f3ff711e0d626f058b273c6be5
Oct 30 20:09:51.818360 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.818341    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"]
Oct 30 20:09:51.821362 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.821334    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:09:51.821442 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.821418    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:09:51.821478 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:51.821464    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:09:52.609455 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:52.609414    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:09:52.739902 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:52.739861    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg" event={"ID":"db939b9a-f127-428e-bc8f-c137346b089c","Type":"ContainerStarted","Data":"1ba53a2fa2b28cf6474b97160b11b7e382aa0629143799d2fb8152372276a945"}
Oct 30 20:09:52.740116 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:52.739923    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg" event={"ID":"db939b9a-f127-428e-bc8f-c137346b089c","Type":"ContainerStarted","Data":"bff7403d2443a6acc9960ae2703cd95bf675a6f3ff711e0d626f058b273c6be5"}
Oct 30 20:09:52.742255 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:52.742219    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr" event={"ID":"a57ba962-389a-4798-a788-cb0dedfa0f5c","Type":"ContainerStarted","Data":"6c53871af6c606bd4fec21f6becc9eebb62a76cb102ff366a0f53a6573b85af8"}
Oct 30 20:09:52.742372 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:52.742269    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr" event={"ID":"a57ba962-389a-4798-a788-cb0dedfa0f5c","Type":"ContainerStarted","Data":"c3e37570b11c4d903de1e9e10fd902e5805d2f4a23503cee95a3112a9b7173cd"}
Oct 30 20:09:52.839075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:52.831229    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg" podStartSLOduration=1.831208933 podStartE2EDuration="1.831208933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:52.773576727 +0000 UTC m=+2271.689183583" watchObservedRunningTime="2026-10-30 20:09:52.831208933 +0000 UTC m=+2271.746815790"
Oct 30 20:09:52.881075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:52.880900    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr" podStartSLOduration=1.880875637 podStartE2EDuration="1.880875637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:52.811190108 +0000 UTC m=+2271.726796965" watchObservedRunningTime="2026-10-30 20:09:52.880875637 +0000 UTC m=+2271.796482494"
Oct 30 20:09:53.435513 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:53.435481    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:09:53.436945 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:53.436917    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 20:09:53.437028 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:53.436974    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr" podUID="a57ba962-389a-4798-a788-cb0dedfa0f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 20:09:53.521200 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:53.521146    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:09:53.522036 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:53.522004    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:09:53.522206 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:53.522096    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg" podUID="db939b9a-f127-428e-bc8f-c137346b089c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 20:09:54.444069 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:54.436448    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 20:09:54.444069 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:54.436508    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr" podUID="a57ba962-389a-4798-a788-cb0dedfa0f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 20:09:54.526074 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:54.517457    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:09:54.526074 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:54.517519    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg" podUID="db939b9a-f127-428e-bc8f-c137346b089c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 20:09:55.435949 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:55.435915    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 20:09:55.436175 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:55.435974    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr" podUID="a57ba962-389a-4798-a788-cb0dedfa0f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 20:09:55.515480 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:55.515439    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:09:55.516128 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:55.516096    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg" podUID="db939b9a-f127-428e-bc8f-c137346b089c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 20:09:56.435567 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:56.435534    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 20:09:56.435808 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:56.435786    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr" podUID="a57ba962-389a-4798-a788-cb0dedfa0f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 20:09:56.520075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:56.519556    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:09:56.520075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:56.519636    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg" podUID="db939b9a-f127-428e-bc8f-c137346b089c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 20:09:57.435436 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:57.435387    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 20:09:57.435619 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:57.435455    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr" podUID="a57ba962-389a-4798-a788-cb0dedfa0f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 20:09:57.522071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:57.521483    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:09:57.522071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:57.521545    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg" podUID="db939b9a-f127-428e-bc8f-c137346b089c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 20:09:58.435380 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:58.435345    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 20:09:58.435547 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:58.435403    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr" podUID="a57ba962-389a-4798-a788-cb0dedfa0f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 20:09:58.516092 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:58.516034    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:09:58.516258 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:58.516124    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg" podUID="db939b9a-f127-428e-bc8f-c137346b089c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 20:09:59.435398 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:59.435358    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 20:09:59.435778 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:59.435431    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr" podUID="a57ba962-389a-4798-a788-cb0dedfa0f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 20:09:59.515561 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:59.515524    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:09:59.515729 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:09:59.515590    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg" podUID="db939b9a-f127-428e-bc8f-c137346b089c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 20:10:00.983565 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:00.983528    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:10:00.983990 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:00.983909    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:10:00.984735 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:00.984712    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-x6hbg"
Oct 30 20:10:01.060158 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:01.060125    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:10:01.060520 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:01.060488    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:10:01.061493 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:01.061472    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:10:04.290668 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:10:04.290627    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:10:10.582798 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:10.581116    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"]
Oct 30 20:10:15.582581 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:15.582537    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc" podUID="b00abc9d-1729-4416-ab87-55e3a0f9ad64" containerName="payload-processing" containerID="cri-o://34f91b6e85d663a7aa36af4f0e327432a2e3a9ba3bc4a1f3adc39ea72481606b" gracePeriod=25
Oct 30 20:10:15.857743 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:15.857717    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:10:15.990061 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:15.990010    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00abc9d-1729-4416-ab87-55e3a0f9ad64-kube-api-access-p2cbz\" (UniqueName: \"kubernetes.io/projected/b00abc9d-1729-4416-ab87-55e3a0f9ad64-kube-api-access-p2cbz\") pod \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\" (UID: \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\") "
Oct 30 20:10:15.990280 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:15.990260    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b00abc9d-1729-4416-ab87-55e3a0f9ad64-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b00abc9d-1729-4416-ab87-55e3a0f9ad64-routing-overlay-volume\") pod \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\" (UID: \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\") "
Oct 30 20:10:15.990355 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:15.990338    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00abc9d-1729-4416-ab87-55e3a0f9ad64-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b00abc9d-1729-4416-ab87-55e3a0f9ad64-provider-credentials\") pod \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\" (UID: \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\") "
Oct 30 20:10:15.990399 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:15.990389    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b00abc9d-1729-4416-ab87-55e3a0f9ad64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b00abc9d-1729-4416-ab87-55e3a0f9ad64-plugins-config-volume\") pod \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\" (UID: \"b00abc9d-1729-4416-ab87-55e3a0f9ad64\") "
Oct 30 20:10:15.990910 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:15.990740    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00abc9d-1729-4416-ab87-55e3a0f9ad64-routing-overlay-volume" pod "b00abc9d-1729-4416-ab87-55e3a0f9ad64" (UID: "b00abc9d-1729-4416-ab87-55e3a0f9ad64"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:15.990910 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:15.990845    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00abc9d-1729-4416-ab87-55e3a0f9ad64-plugins-config-volume" pod "b00abc9d-1729-4416-ab87-55e3a0f9ad64" (UID: "b00abc9d-1729-4416-ab87-55e3a0f9ad64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:15.991125 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:15.991038    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b00abc9d-1729-4416-ab87-55e3a0f9ad64-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:15.991125 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:15.991091    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b00abc9d-1729-4416-ab87-55e3a0f9ad64-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:15.992543 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:15.992512    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00abc9d-1729-4416-ab87-55e3a0f9ad64-kube-api-access-p2cbz" pod "b00abc9d-1729-4416-ab87-55e3a0f9ad64" (UID: "b00abc9d-1729-4416-ab87-55e3a0f9ad64"). InnerVolumeSpecName "kube-api-access-p2cbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:15.992656 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:15.992581    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00abc9d-1729-4416-ab87-55e3a0f9ad64-provider-credentials" pod "b00abc9d-1729-4416-ab87-55e3a0f9ad64" (UID: "b00abc9d-1729-4416-ab87-55e3a0f9ad64"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:16.007213 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:16.007189    6030 generic.go:356] "Generic (PLEG): container finished" podID="b00abc9d-1729-4416-ab87-55e3a0f9ad64" containerID="34f91b6e85d663a7aa36af4f0e327432a2e3a9ba3bc4a1f3adc39ea72481606b" exitCode=0
Oct 30 20:10:16.007340 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:16.007236    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"
Oct 30 20:10:16.007340 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:16.007278    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc" event={"ID":"b00abc9d-1729-4416-ab87-55e3a0f9ad64","Type":"ContainerDied","Data":"34f91b6e85d663a7aa36af4f0e327432a2e3a9ba3bc4a1f3adc39ea72481606b"}
Oct 30 20:10:16.007467 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:16.007349    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc" event={"ID":"b00abc9d-1729-4416-ab87-55e3a0f9ad64","Type":"ContainerDied","Data":"d74c6ec1bd7d94aabf6633efc589587a41676c583843c2c3c062d45801d95ef2"}
Oct 30 20:10:16.007467 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:16.007376    6030 scope.go:122] "RemoveContainer" containerID="34f91b6e85d663a7aa36af4f0e327432a2e3a9ba3bc4a1f3adc39ea72481606b"
Oct 30 20:10:16.022438 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:16.022418    6030 scope.go:122] "RemoveContainer" containerID="34f91b6e85d663a7aa36af4f0e327432a2e3a9ba3bc4a1f3adc39ea72481606b"
Oct 30 20:10:16.022729 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:10:16.022706    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f91b6e85d663a7aa36af4f0e327432a2e3a9ba3bc4a1f3adc39ea72481606b\": container with ID starting with 34f91b6e85d663a7aa36af4f0e327432a2e3a9ba3bc4a1f3adc39ea72481606b not found: ID does not exist" containerID="34f91b6e85d663a7aa36af4f0e327432a2e3a9ba3bc4a1f3adc39ea72481606b"
Oct 30 20:10:16.022793 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:16.022742    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f91b6e85d663a7aa36af4f0e327432a2e3a9ba3bc4a1f3adc39ea72481606b"} err="failed to get container status \"34f91b6e85d663a7aa36af4f0e327432a2e3a9ba3bc4a1f3adc39ea72481606b\": rpc error: code = NotFound desc = could not find container \"34f91b6e85d663a7aa36af4f0e327432a2e3a9ba3bc4a1f3adc39ea72481606b\": container with ID starting with 34f91b6e85d663a7aa36af4f0e327432a2e3a9ba3bc4a1f3adc39ea72481606b not found: ID does not exist"
Oct 30 20:10:16.042450 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:16.042428    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"]
Oct 30 20:10:16.045476 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:16.045453    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-5778x9jsc"]
Oct 30 20:10:16.092950 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:16.092880    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b00abc9d-1729-4416-ab87-55e3a0f9ad64-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:16.092950 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:16.092915    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2cbz\" (UniqueName: \"kubernetes.io/projected/b00abc9d-1729-4416-ab87-55e3a0f9ad64-kube-api-access-p2cbz\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:17.203138 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:10:17.203108    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b00abc9d-1729-4416-ab87-55e3a0f9ad64" path="/var/lib/kubelet/pods/b00abc9d-1729-4416-ab87-55e3a0f9ad64/volumes"
Oct 30 20:10:34.289362 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:10:34.289322    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:11:04.290540 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:11:04.290498    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:11:34.290378 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:11:34.290339    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:12:04.327618 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:12:04.327576    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:12:16.794850 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:16.794812    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"]
Oct 30 20:12:16.795485 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:16.795077    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd" podUID="618c01b7-db1f-47a4-be57-aa701de1a286" containerName="discovery" containerID="cri-o://48520e40bd56963a5442ca72c5064573016b08a05765f1ed0edc631b96431407" gracePeriod=30
Oct 30 20:12:17.033644 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.032497    6030 generic.go:356] "Generic (PLEG): container finished" podID="618c01b7-db1f-47a4-be57-aa701de1a286" containerID="48520e40bd56963a5442ca72c5064573016b08a05765f1ed0edc631b96431407" exitCode=0
Oct 30 20:12:17.033644 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.033566    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd" event={"ID":"618c01b7-db1f-47a4-be57-aa701de1a286","Type":"ContainerDied","Data":"48520e40bd56963a5442ca72c5064573016b08a05765f1ed0edc631b96431407"}
Oct 30 20:12:17.100741 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.100714    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 20:12:17.134171 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.134138    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618c01b7-db1f-47a4-be57-aa701de1a286-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/618c01b7-db1f-47a4-be57-aa701de1a286-local-certs\") pod \"618c01b7-db1f-47a4-be57-aa701de1a286\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") "
Oct 30 20:12:17.134347 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.134214    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/618c01b7-db1f-47a4-be57-aa701de1a286-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/618c01b7-db1f-47a4-be57-aa701de1a286-istio-csr-ca-configmap\") pod \"618c01b7-db1f-47a4-be57-aa701de1a286\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") "
Oct 30 20:12:17.134347 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.134340    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-istio-kubeconfig\") pod \"618c01b7-db1f-47a4-be57-aa701de1a286\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") "
Oct 30 20:12:17.134475 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.134425    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/618c01b7-db1f-47a4-be57-aa701de1a286-kube-api-access-wc6d9\" (UniqueName: \"kubernetes.io/projected/618c01b7-db1f-47a4-be57-aa701de1a286-kube-api-access-wc6d9\") pod \"618c01b7-db1f-47a4-be57-aa701de1a286\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") "
Oct 30 20:12:17.134611 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.134580    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-istio-csr-dns-cert\") pod \"618c01b7-db1f-47a4-be57-aa701de1a286\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") "
Oct 30 20:12:17.134910 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.134640    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/618c01b7-db1f-47a4-be57-aa701de1a286-istio-token\" (UniqueName: \"kubernetes.io/projected/618c01b7-db1f-47a4-be57-aa701de1a286-istio-token\") pod \"618c01b7-db1f-47a4-be57-aa701de1a286\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") "
Oct 30 20:12:17.134910 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.134712    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-cacerts\" (UniqueName: \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-cacerts\") pod \"618c01b7-db1f-47a4-be57-aa701de1a286\" (UID: \"618c01b7-db1f-47a4-be57-aa701de1a286\") "
Oct 30 20:12:17.134910 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.134779    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/618c01b7-db1f-47a4-be57-aa701de1a286-istio-csr-ca-configmap" pod "618c01b7-db1f-47a4-be57-aa701de1a286" (UID: "618c01b7-db1f-47a4-be57-aa701de1a286"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:17.136369 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.135283    6030 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/618c01b7-db1f-47a4-be57-aa701de1a286-istio-csr-ca-configmap\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:17.138119 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.138035    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-istio-csr-dns-cert" pod "618c01b7-db1f-47a4-be57-aa701de1a286" (UID: "618c01b7-db1f-47a4-be57-aa701de1a286"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:17.138255 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.138112    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-cacerts" pod "618c01b7-db1f-47a4-be57-aa701de1a286" (UID: "618c01b7-db1f-47a4-be57-aa701de1a286"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:17.143992 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.143601    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618c01b7-db1f-47a4-be57-aa701de1a286-kube-api-access-wc6d9" pod "618c01b7-db1f-47a4-be57-aa701de1a286" (UID: "618c01b7-db1f-47a4-be57-aa701de1a286"). InnerVolumeSpecName "kube-api-access-wc6d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:17.143992 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.143676    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-istio-kubeconfig" pod "618c01b7-db1f-47a4-be57-aa701de1a286" (UID: "618c01b7-db1f-47a4-be57-aa701de1a286"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:17.143992 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.143696    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618c01b7-db1f-47a4-be57-aa701de1a286-local-certs" pod "618c01b7-db1f-47a4-be57-aa701de1a286" (UID: "618c01b7-db1f-47a4-be57-aa701de1a286"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:17.143992 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.143851    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618c01b7-db1f-47a4-be57-aa701de1a286-istio-token" pod "618c01b7-db1f-47a4-be57-aa701de1a286" (UID: "618c01b7-db1f-47a4-be57-aa701de1a286"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:17.237032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.237000    6030 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/618c01b7-db1f-47a4-be57-aa701de1a286-local-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:17.237238 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.237035    6030 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-istio-kubeconfig\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:17.237238 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.237092    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc6d9\" (UniqueName: \"kubernetes.io/projected/618c01b7-db1f-47a4-be57-aa701de1a286-kube-api-access-wc6d9\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:17.237238 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.237110    6030 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-istio-csr-dns-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:17.237238 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.237124    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618c01b7-db1f-47a4-be57-aa701de1a286-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:17.237238 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:17.237139    6030 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/618c01b7-db1f-47a4-be57-aa701de1a286-cacerts\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:18.042091 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:18.042028    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"
Oct 30 20:12:18.042091 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:18.042073    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd" event={"ID":"618c01b7-db1f-47a4-be57-aa701de1a286","Type":"ContainerDied","Data":"190412ce73d2effadbbf3e66134958d87df3811b2f2739413407ad119d5ce315"}
Oct 30 20:12:18.042604 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:18.042140    6030 scope.go:122] "RemoveContainer" containerID="48520e40bd56963a5442ca72c5064573016b08a05765f1ed0edc631b96431407"
Oct 30 20:12:18.073098 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:18.072225    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"]
Oct 30 20:12:18.075719 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:18.075696    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p8jcd"]
Oct 30 20:12:19.204172 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:19.204144    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="618c01b7-db1f-47a4-be57-aa701de1a286" path="/var/lib/kubelet/pods/618c01b7-db1f-47a4-be57-aa701de1a286/volumes"
Oct 30 20:12:20.430762 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:20.430720    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"]
Oct 30 20:12:25.432356 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:25.432312    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj" podUID="b4a22544-17f5-4072-b5cb-876ed3f2f80b" containerName="payload-processing" containerID="cri-o://a7959cf952a8448ca154fa795f13ff0b3f2a99dd145186e21cc60bfcb3077296" gracePeriod=25
Oct 30 20:12:25.736185 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:25.736156    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:12:25.857588 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:25.857555    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a22544-17f5-4072-b5cb-876ed3f2f80b-kube-api-access-2s4xf\" (UniqueName: \"kubernetes.io/projected/b4a22544-17f5-4072-b5cb-876ed3f2f80b-kube-api-access-2s4xf\") pod \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\" (UID: \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\") "
Oct 30 20:12:25.857781 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:25.857669    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a22544-17f5-4072-b5cb-876ed3f2f80b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4a22544-17f5-4072-b5cb-876ed3f2f80b-routing-overlay-volume\") pod \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\" (UID: \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\") "
Oct 30 20:12:25.857841 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:25.857804    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a22544-17f5-4072-b5cb-876ed3f2f80b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a22544-17f5-4072-b5cb-876ed3f2f80b-plugins-config-volume\") pod \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\" (UID: \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\") "
Oct 30 20:12:25.857949 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:25.857865    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a22544-17f5-4072-b5cb-876ed3f2f80b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4a22544-17f5-4072-b5cb-876ed3f2f80b-provider-credentials\") pod \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\" (UID: \"b4a22544-17f5-4072-b5cb-876ed3f2f80b\") "
Oct 30 20:12:25.858171 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:25.858113    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a22544-17f5-4072-b5cb-876ed3f2f80b-routing-overlay-volume" pod "b4a22544-17f5-4072-b5cb-876ed3f2f80b" (UID: "b4a22544-17f5-4072-b5cb-876ed3f2f80b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:25.858406 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:25.858376    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a22544-17f5-4072-b5cb-876ed3f2f80b-plugins-config-volume" pod "b4a22544-17f5-4072-b5cb-876ed3f2f80b" (UID: "b4a22544-17f5-4072-b5cb-876ed3f2f80b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:25.858534 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:25.858511    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a22544-17f5-4072-b5cb-876ed3f2f80b-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.858603 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:25.858542    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4a22544-17f5-4072-b5cb-876ed3f2f80b-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.860202 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:25.860182    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a22544-17f5-4072-b5cb-876ed3f2f80b-kube-api-access-2s4xf" pod "b4a22544-17f5-4072-b5cb-876ed3f2f80b" (UID: "b4a22544-17f5-4072-b5cb-876ed3f2f80b"). InnerVolumeSpecName "kube-api-access-2s4xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:25.860310 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:25.860289    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a22544-17f5-4072-b5cb-876ed3f2f80b-provider-credentials" pod "b4a22544-17f5-4072-b5cb-876ed3f2f80b" (UID: "b4a22544-17f5-4072-b5cb-876ed3f2f80b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:25.960274 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:25.960195    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4a22544-17f5-4072-b5cb-876ed3f2f80b-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.960274 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:25.960225    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s4xf\" (UniqueName: \"kubernetes.io/projected/b4a22544-17f5-4072-b5cb-876ed3f2f80b-kube-api-access-2s4xf\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:26.104946 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:26.104915    6030 generic.go:356] "Generic (PLEG): container finished" podID="b4a22544-17f5-4072-b5cb-876ed3f2f80b" containerID="a7959cf952a8448ca154fa795f13ff0b3f2a99dd145186e21cc60bfcb3077296" exitCode=0
Oct 30 20:12:26.105155 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:26.104981    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"
Oct 30 20:12:26.105155 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:26.104997    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj" event={"ID":"b4a22544-17f5-4072-b5cb-876ed3f2f80b","Type":"ContainerDied","Data":"a7959cf952a8448ca154fa795f13ff0b3f2a99dd145186e21cc60bfcb3077296"}
Oct 30 20:12:26.105155 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:26.105100    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj" event={"ID":"b4a22544-17f5-4072-b5cb-876ed3f2f80b","Type":"ContainerDied","Data":"86ac4e81b936bfa7db21892beaa8ec0394b699b676a7133ed306098f4e6778bd"}
Oct 30 20:12:26.105155 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:26.105124    6030 scope.go:122] "RemoveContainer" containerID="a7959cf952a8448ca154fa795f13ff0b3f2a99dd145186e21cc60bfcb3077296"
Oct 30 20:12:26.116228 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:26.116210    6030 scope.go:122] "RemoveContainer" containerID="a7959cf952a8448ca154fa795f13ff0b3f2a99dd145186e21cc60bfcb3077296"
Oct 30 20:12:26.116474 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:12:26.116453    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7959cf952a8448ca154fa795f13ff0b3f2a99dd145186e21cc60bfcb3077296\": container with ID starting with a7959cf952a8448ca154fa795f13ff0b3f2a99dd145186e21cc60bfcb3077296 not found: ID does not exist" containerID="a7959cf952a8448ca154fa795f13ff0b3f2a99dd145186e21cc60bfcb3077296"
Oct 30 20:12:26.116539 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:26.116485    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7959cf952a8448ca154fa795f13ff0b3f2a99dd145186e21cc60bfcb3077296"} err="failed to get container status \"a7959cf952a8448ca154fa795f13ff0b3f2a99dd145186e21cc60bfcb3077296\": rpc error: code = NotFound desc = could not find container \"a7959cf952a8448ca154fa795f13ff0b3f2a99dd145186e21cc60bfcb3077296\": container with ID starting with a7959cf952a8448ca154fa795f13ff0b3f2a99dd145186e21cc60bfcb3077296 not found: ID does not exist"
Oct 30 20:12:26.134207 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:26.134182    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"]
Oct 30 20:12:26.139830 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:26.139810    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-689cpfxgj"]
Oct 30 20:12:27.203341 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:27.203305    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a22544-17f5-4072-b5cb-876ed3f2f80b" path="/var/lib/kubelet/pods/b4a22544-17f5-4072-b5cb-876ed3f2f80b/volumes"
Oct 30 20:12:30.717971 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.717930    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"]
Oct 30 20:12:30.718647 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.718630    6030 cpu_manager.go:414] "containerMap: removing container" podUID="b00abc9d-1729-4416-ab87-55e3a0f9ad64" containerName="payload-processing"
Oct 30 20:12:30.718647 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.718647    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00abc9d-1729-4416-ab87-55e3a0f9ad64" containerName="payload-processing"
Oct 30 20:12:30.718797 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.718673    6030 cpu_manager.go:414] "containerMap: removing container" podUID="618c01b7-db1f-47a4-be57-aa701de1a286" containerName="discovery"
Oct 30 20:12:30.718797 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.718679    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="618c01b7-db1f-47a4-be57-aa701de1a286" containerName="discovery"
Oct 30 20:12:30.718797 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.718701    6030 cpu_manager.go:414] "containerMap: removing container" podUID="b4a22544-17f5-4072-b5cb-876ed3f2f80b" containerName="payload-processing"
Oct 30 20:12:30.718797 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.718707    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a22544-17f5-4072-b5cb-876ed3f2f80b" containerName="payload-processing"
Oct 30 20:12:30.719020 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.718802    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00abc9d-1729-4416-ab87-55e3a0f9ad64" containerName="payload-processing"
Oct 30 20:12:30.719020 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.718814    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="618c01b7-db1f-47a4-be57-aa701de1a286" containerName="discovery"
Oct 30 20:12:30.719020 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.718824    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a22544-17f5-4072-b5cb-876ed3f2f80b" containerName="payload-processing"
Oct 30 20:12:30.724113 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.724090    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:30.732497 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.732475    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"routing-overlay\""
Oct 30 20:12:30.732656 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.732499    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"kube-root-ca.crt\""
Oct 30 20:12:30.733344 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.733323    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"]
Oct 30 20:12:30.734032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.733998    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"openshift-service-ca.crt\""
Oct 30 20:12:30.734193 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.734086    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d97cc8\""
Oct 30 20:12:30.734193 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.734097    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"payload-processing-external-model-e2e--151804b4-dockercfg-xxgch\""
Oct 30 20:12:30.734193 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.734184    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"agc-cred-52dc941f181c\""
Oct 30 20:12:30.829359 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.829325    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea94ede7-d7c9-43c3-b136-6cacb5137565-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w\" (UID: \"ea94ede7-d7c9-43c3-b136-6cacb5137565\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:30.829359 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.829362    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea94ede7-d7c9-43c3-b136-6cacb5137565-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w\" (UID: \"ea94ede7-d7c9-43c3-b136-6cacb5137565\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:30.829555 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.829477    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea94ede7-d7c9-43c3-b136-6cacb5137565-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w\" (UID: \"ea94ede7-d7c9-43c3-b136-6cacb5137565\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:30.829695 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.829676    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbkk8\" (UniqueName: \"kubernetes.io/projected/ea94ede7-d7c9-43c3-b136-6cacb5137565-kube-api-access-jbkk8\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w\" (UID: \"ea94ede7-d7c9-43c3-b136-6cacb5137565\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:30.954028 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:30.943114    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbkk8\" (UniqueName: \"kubernetes.io/projected/ea94ede7-d7c9-43c3-b136-6cacb5137565-kube-api-access-jbkk8\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w\" (UID: \"ea94ede7-d7c9-43c3-b136-6cacb5137565\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:31.003258 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.003150    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea94ede7-d7c9-43c3-b136-6cacb5137565-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w\" (UID: \"ea94ede7-d7c9-43c3-b136-6cacb5137565\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:31.003577 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.003511    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea94ede7-d7c9-43c3-b136-6cacb5137565-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w\" (UID: \"ea94ede7-d7c9-43c3-b136-6cacb5137565\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:31.003741 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.003661    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea94ede7-d7c9-43c3-b136-6cacb5137565-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w\" (UID: \"ea94ede7-d7c9-43c3-b136-6cacb5137565\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:31.004787 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.004762    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea94ede7-d7c9-43c3-b136-6cacb5137565-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w\" (UID: \"ea94ede7-d7c9-43c3-b136-6cacb5137565\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:31.005409 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.005314    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea94ede7-d7c9-43c3-b136-6cacb5137565-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w\" (UID: \"ea94ede7-d7c9-43c3-b136-6cacb5137565\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:31.032545 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.030718    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbkk8\" (UniqueName: \"kubernetes.io/projected/ea94ede7-d7c9-43c3-b136-6cacb5137565-kube-api-access-jbkk8\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w\" (UID: \"ea94ede7-d7c9-43c3-b136-6cacb5137565\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:31.032545 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.031228    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea94ede7-d7c9-43c3-b136-6cacb5137565-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w\" (UID: \"ea94ede7-d7c9-43c3-b136-6cacb5137565\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:31.045514 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.043306    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:31.153229 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.153194    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"]
Oct 30 20:12:31.167217 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.167183    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"]
Oct 30 20:12:31.167439 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.167423    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:31.174642 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.174314    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"agc-cred-6e22e9b2ae2b\""
Oct 30 20:12:31.174642 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.174426    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c9bce8\""
Oct 30 20:12:31.174642 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.174613    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"openshift-service-ca.crt\""
Oct 30 20:12:31.175282 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.175105    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"routing-overlay\""
Oct 30 20:12:31.175282 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.175186    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"payload-processing-external-model-e2e--d955aedb-dockercfg-8b69h\""
Oct 30 20:12:31.175282 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.175254    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"kube-root-ca.crt\""
Oct 30 20:12:31.207607 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.207569    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/220d793a-4370-4d29-a3d1-7fe3903afca8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6\" (UID: \"220d793a-4370-4d29-a3d1-7fe3903afca8\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:31.207864 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.207827    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/220d793a-4370-4d29-a3d1-7fe3903afca8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6\" (UID: \"220d793a-4370-4d29-a3d1-7fe3903afca8\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:31.207952 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.207881    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/220d793a-4370-4d29-a3d1-7fe3903afca8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6\" (UID: \"220d793a-4370-4d29-a3d1-7fe3903afca8\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:31.208030 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.208014    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvgr5\" (UniqueName: \"kubernetes.io/projected/220d793a-4370-4d29-a3d1-7fe3903afca8-kube-api-access-tvgr5\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6\" (UID: \"220d793a-4370-4d29-a3d1-7fe3903afca8\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:31.309624 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.309531    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/220d793a-4370-4d29-a3d1-7fe3903afca8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6\" (UID: \"220d793a-4370-4d29-a3d1-7fe3903afca8\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:31.309806 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.309710    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/220d793a-4370-4d29-a3d1-7fe3903afca8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6\" (UID: \"220d793a-4370-4d29-a3d1-7fe3903afca8\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:31.309806 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.309740    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/220d793a-4370-4d29-a3d1-7fe3903afca8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6\" (UID: \"220d793a-4370-4d29-a3d1-7fe3903afca8\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:31.309933 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.309860    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvgr5\" (UniqueName: \"kubernetes.io/projected/220d793a-4370-4d29-a3d1-7fe3903afca8-kube-api-access-tvgr5\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6\" (UID: \"220d793a-4370-4d29-a3d1-7fe3903afca8\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:31.310416 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.310381    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/220d793a-4370-4d29-a3d1-7fe3903afca8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6\" (UID: \"220d793a-4370-4d29-a3d1-7fe3903afca8\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:31.310715 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.310449    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/220d793a-4370-4d29-a3d1-7fe3903afca8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6\" (UID: \"220d793a-4370-4d29-a3d1-7fe3903afca8\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:31.312392 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.312370    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/220d793a-4370-4d29-a3d1-7fe3903afca8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6\" (UID: \"220d793a-4370-4d29-a3d1-7fe3903afca8\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:31.326174 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.326149    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvgr5\" (UniqueName: \"kubernetes.io/projected/220d793a-4370-4d29-a3d1-7fe3903afca8-kube-api-access-tvgr5\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6\" (UID: \"220d793a-4370-4d29-a3d1-7fe3903afca8\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:31.348039 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.348014    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"]
Oct 30 20:12:31.349420 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:12:31.349396    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea94ede7_d7c9_43c3_b136_6cacb5137565.slice/crio-70f2bf41c86809a219b67b16fab781223144c2559c9ab42f8d4cd23e555f7bf2 WatchSource:0}: Error finding container 70f2bf41c86809a219b67b16fab781223144c2559c9ab42f8d4cd23e555f7bf2: Status 404 returned error can't find the container with id 70f2bf41c86809a219b67b16fab781223144c2559c9ab42f8d4cd23e555f7bf2
Oct 30 20:12:31.491464 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.491432    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:31.802515 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:31.802489    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"]
Oct 30 20:12:31.804910 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:12:31.804883    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220d793a_4370_4d29_a3d1_7fe3903afca8.slice/crio-a588e53d3fcf4022c7e926de36ff857ef7985153b79bbff0196510e8912471c9 WatchSource:0}: Error finding container a588e53d3fcf4022c7e926de36ff857ef7985153b79bbff0196510e8912471c9: Status 404 returned error can't find the container with id a588e53d3fcf4022c7e926de36ff857ef7985153b79bbff0196510e8912471c9
Oct 30 20:12:32.167721 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.167687    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w" event={"ID":"ea94ede7-d7c9-43c3-b136-6cacb5137565","Type":"ContainerStarted","Data":"75949f44bdd997a0febe659a765990c426577bbd1979336a01af5aa0437ba26e"}
Oct 30 20:12:32.167887 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.167749    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:32.167887 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.167768    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w" event={"ID":"ea94ede7-d7c9-43c3-b136-6cacb5137565","Type":"ContainerStarted","Data":"70f2bf41c86809a219b67b16fab781223144c2559c9ab42f8d4cd23e555f7bf2"}
Oct 30 20:12:32.169427 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.169401    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6" event={"ID":"220d793a-4370-4d29-a3d1-7fe3903afca8","Type":"ContainerStarted","Data":"6142b726d71e8e33d8e5a2a020f77f1538a3fbc769439321aed66f8da17f80cf"}
Oct 30 20:12:32.169554 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.169444    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6" event={"ID":"220d793a-4370-4d29-a3d1-7fe3903afca8","Type":"ContainerStarted","Data":"a588e53d3fcf4022c7e926de36ff857ef7985153b79bbff0196510e8912471c9"}
Oct 30 20:12:32.169554 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.169547    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:32.199810 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.199756    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w" podStartSLOduration=2.19973918 podStartE2EDuration="2.19973918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:32.197157162 +0000 UTC m=+2431.112764019" watchObservedRunningTime="2026-10-30 20:12:32.19973918 +0000 UTC m=+2431.115346038"
Oct 30 20:12:32.241992 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.241934    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6" podStartSLOduration=1.24192089 podStartE2EDuration="1.24192089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:32.230725277 +0000 UTC m=+2431.146332133" watchObservedRunningTime="2026-10-30 20:12:32.24192089 +0000 UTC m=+2431.157527746"
Oct 30 20:12:32.273101 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.273071    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"]
Oct 30 20:12:32.277848 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.277821    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:12:32.291299 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.291279    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"agc-cred-f4868d300b41\""
Oct 30 20:12:32.302057 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.302017    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"]
Oct 30 20:12:32.326725 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.326686    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0378ab28-33fd-4012-8a51-239e35ef201d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2\" (UID: \"0378ab28-33fd-4012-8a51-239e35ef201d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:12:32.326892 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.326866    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0378ab28-33fd-4012-8a51-239e35ef201d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2\" (UID: \"0378ab28-33fd-4012-8a51-239e35ef201d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:12:32.327004 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.326898    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grmpx\" (UniqueName: \"kubernetes.io/projected/0378ab28-33fd-4012-8a51-239e35ef201d-kube-api-access-grmpx\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2\" (UID: \"0378ab28-33fd-4012-8a51-239e35ef201d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:12:32.327004 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.326965    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0378ab28-33fd-4012-8a51-239e35ef201d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2\" (UID: \"0378ab28-33fd-4012-8a51-239e35ef201d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:12:32.428350 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.428245    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0378ab28-33fd-4012-8a51-239e35ef201d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2\" (UID: \"0378ab28-33fd-4012-8a51-239e35ef201d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:12:32.428350 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.428310    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grmpx\" (UniqueName: \"kubernetes.io/projected/0378ab28-33fd-4012-8a51-239e35ef201d-kube-api-access-grmpx\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2\" (UID: \"0378ab28-33fd-4012-8a51-239e35ef201d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:12:32.428592 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.428369    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0378ab28-33fd-4012-8a51-239e35ef201d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2\" (UID: \"0378ab28-33fd-4012-8a51-239e35ef201d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:12:32.428592 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.428557    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0378ab28-33fd-4012-8a51-239e35ef201d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2\" (UID: \"0378ab28-33fd-4012-8a51-239e35ef201d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:12:32.429184 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.429146    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0378ab28-33fd-4012-8a51-239e35ef201d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2\" (UID: \"0378ab28-33fd-4012-8a51-239e35ef201d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:12:32.429815 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.429779    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0378ab28-33fd-4012-8a51-239e35ef201d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2\" (UID: \"0378ab28-33fd-4012-8a51-239e35ef201d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:12:32.431566 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.431542    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0378ab28-33fd-4012-8a51-239e35ef201d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2\" (UID: \"0378ab28-33fd-4012-8a51-239e35ef201d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:12:32.453701 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.453669    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grmpx\" (UniqueName: \"kubernetes.io/projected/0378ab28-33fd-4012-8a51-239e35ef201d-kube-api-access-grmpx\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2\" (UID: \"0378ab28-33fd-4012-8a51-239e35ef201d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:12:32.593788 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.593757    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:12:32.843087 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.842994    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"]
Oct 30 20:12:32.848746 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.848725    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:12:32.856932 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.856906    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"agc-cred-5b9d0f0966be\""
Oct 30 20:12:32.861447 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.860524    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"]
Oct 30 20:12:32.882784 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.882747    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"]
Oct 30 20:12:32.883216 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:12:32.883192    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0378ab28_33fd_4012_8a51_239e35ef201d.slice/crio-05617c22ccbae17f51396c097a4566374e92817c2cae83d1a1c9bb4c46a87942 WatchSource:0}: Error finding container 05617c22ccbae17f51396c097a4566374e92817c2cae83d1a1c9bb4c46a87942: Status 404 returned error can't find the container with id 05617c22ccbae17f51396c097a4566374e92817c2cae83d1a1c9bb4c46a87942
Oct 30 20:12:32.936426 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.936401    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g\" (UID: \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:12:32.936768 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.936732    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g\" (UID: \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:12:32.936876 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.936777    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7wt\" (UniqueName: \"kubernetes.io/projected/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-kube-api-access-7d7wt\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g\" (UID: \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:12:32.936876 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:32.936836    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g\" (UID: \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:12:33.037651 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:33.037620    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g\" (UID: \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:12:33.037837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:33.037658    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7wt\" (UniqueName: \"kubernetes.io/projected/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-kube-api-access-7d7wt\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g\" (UID: \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:12:33.037837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:33.037685    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g\" (UID: \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:12:33.037837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:33.037726    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g\" (UID: \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:12:33.038543 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:33.038493    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g\" (UID: \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:12:33.038670 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:33.038600    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g\" (UID: \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:12:33.040677 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:33.040659    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g\" (UID: \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:12:33.052071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:33.052031    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7wt\" (UniqueName: \"kubernetes.io/projected/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-kube-api-access-7d7wt\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g\" (UID: \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:12:33.167800 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:33.167769    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:12:33.182375 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:33.182341    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2" event={"ID":"0378ab28-33fd-4012-8a51-239e35ef201d","Type":"ContainerStarted","Data":"9c5733d4c45cdcff091caa4b32c7a4a6ed2f3310ad4667ca4358e51e1043562a"}
Oct 30 20:12:33.182507 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:33.182408    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2" event={"ID":"0378ab28-33fd-4012-8a51-239e35ef201d","Type":"ContainerStarted","Data":"05617c22ccbae17f51396c097a4566374e92817c2cae83d1a1c9bb4c46a87942"}
Oct 30 20:12:33.182507 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:33.182442    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:12:33.209095 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:33.209027    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2" podStartSLOduration=1.20901306 podStartE2EDuration="1.20901306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:33.204880613 +0000 UTC m=+2432.120487470" watchObservedRunningTime="2026-10-30 20:12:33.20901306 +0000 UTC m=+2432.124619915"
Oct 30 20:12:33.462469 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:33.462444    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"]
Oct 30 20:12:33.464637 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:12:33.464609    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3c0d95_564b_4989_bc2e_0e1ad394f07f.slice/crio-5c7182b476ef566552cf89d402f8b7ceac2f3d6e1ee8a8ba767e2d90aeffa217 WatchSource:0}: Error finding container 5c7182b476ef566552cf89d402f8b7ceac2f3d6e1ee8a8ba767e2d90aeffa217: Status 404 returned error can't find the container with id 5c7182b476ef566552cf89d402f8b7ceac2f3d6e1ee8a8ba767e2d90aeffa217
Oct 30 20:12:34.191967 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:34.191919    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g" event={"ID":"6b3c0d95-564b-4989-bc2e-0e1ad394f07f","Type":"ContainerStarted","Data":"891a520be807e304689bdf9a24ad8b216fce4a4e0467ad8d15c01d7cde0190a5"}
Oct 30 20:12:34.192438 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:34.191994    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:12:34.192438 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:34.192012    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g" event={"ID":"6b3c0d95-564b-4989-bc2e-0e1ad394f07f","Type":"ContainerStarted","Data":"5c7182b476ef566552cf89d402f8b7ceac2f3d6e1ee8a8ba767e2d90aeffa217"}
Oct 30 20:12:34.224322 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:34.224278    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g" podStartSLOduration=2.224264512 podStartE2EDuration="2.224264512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:34.219985609 +0000 UTC m=+2433.135592466" watchObservedRunningTime="2026-10-30 20:12:34.224264512 +0000 UTC m=+2433.139871394"
Oct 30 20:12:34.291585 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:12:34.291552    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:12:43.185109 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:43.185016    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:43.185514 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:43.185113    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:44.195865 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:44.195836    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:12:44.258029 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:44.257993    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"]
Oct 30 20:12:45.203142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:45.203113    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:12:45.269981 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:45.269945    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"]
Oct 30 20:12:49.258903 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:49.258863    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w" podUID="ea94ede7-d7c9-43c3-b136-6cacb5137565" containerName="payload-processing" containerID="cri-o://75949f44bdd997a0febe659a765990c426577bbd1979336a01af5aa0437ba26e" gracePeriod=25
Oct 30 20:12:49.533135 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:49.533111    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:49.661062 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:49.661011    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea94ede7-d7c9-43c3-b136-6cacb5137565-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea94ede7-d7c9-43c3-b136-6cacb5137565-plugins-config-volume\") pod \"ea94ede7-d7c9-43c3-b136-6cacb5137565\" (UID: \"ea94ede7-d7c9-43c3-b136-6cacb5137565\") "
Oct 30 20:12:49.661251 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:49.661122    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea94ede7-d7c9-43c3-b136-6cacb5137565-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea94ede7-d7c9-43c3-b136-6cacb5137565-provider-credentials\") pod \"ea94ede7-d7c9-43c3-b136-6cacb5137565\" (UID: \"ea94ede7-d7c9-43c3-b136-6cacb5137565\") "
Oct 30 20:12:49.661251 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:49.661187    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea94ede7-d7c9-43c3-b136-6cacb5137565-kube-api-access-jbkk8\" (UniqueName: \"kubernetes.io/projected/ea94ede7-d7c9-43c3-b136-6cacb5137565-kube-api-access-jbkk8\") pod \"ea94ede7-d7c9-43c3-b136-6cacb5137565\" (UID: \"ea94ede7-d7c9-43c3-b136-6cacb5137565\") "
Oct 30 20:12:49.661393 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:49.661304    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea94ede7-d7c9-43c3-b136-6cacb5137565-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea94ede7-d7c9-43c3-b136-6cacb5137565-routing-overlay-volume\") pod \"ea94ede7-d7c9-43c3-b136-6cacb5137565\" (UID: \"ea94ede7-d7c9-43c3-b136-6cacb5137565\") "
Oct 30 20:12:49.661393 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:49.661375    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea94ede7-d7c9-43c3-b136-6cacb5137565-plugins-config-volume" pod "ea94ede7-d7c9-43c3-b136-6cacb5137565" (UID: "ea94ede7-d7c9-43c3-b136-6cacb5137565"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:49.661660 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:49.661636    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea94ede7-d7c9-43c3-b136-6cacb5137565-routing-overlay-volume" pod "ea94ede7-d7c9-43c3-b136-6cacb5137565" (UID: "ea94ede7-d7c9-43c3-b136-6cacb5137565"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:49.661986 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:49.661928    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea94ede7-d7c9-43c3-b136-6cacb5137565-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:49.661986 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:49.661948    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea94ede7-d7c9-43c3-b136-6cacb5137565-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:49.663642 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:49.663616    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea94ede7-d7c9-43c3-b136-6cacb5137565-kube-api-access-jbkk8" pod "ea94ede7-d7c9-43c3-b136-6cacb5137565" (UID: "ea94ede7-d7c9-43c3-b136-6cacb5137565"). InnerVolumeSpecName "kube-api-access-jbkk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:49.663642 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:49.663621    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea94ede7-d7c9-43c3-b136-6cacb5137565-provider-credentials" pod "ea94ede7-d7c9-43c3-b136-6cacb5137565" (UID: "ea94ede7-d7c9-43c3-b136-6cacb5137565"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:49.762824 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:49.762737    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea94ede7-d7c9-43c3-b136-6cacb5137565-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:49.762824 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:49.762772    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbkk8\" (UniqueName: \"kubernetes.io/projected/ea94ede7-d7c9-43c3-b136-6cacb5137565-kube-api-access-jbkk8\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.271091 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.271017    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6" podUID="220d793a-4370-4d29-a3d1-7fe3903afca8" containerName="payload-processing" containerID="cri-o://6142b726d71e8e33d8e5a2a020f77f1538a3fbc769439321aed66f8da17f80cf" gracePeriod=25
Oct 30 20:12:50.324284 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.324254    6030 generic.go:356] "Generic (PLEG): container finished" podID="ea94ede7-d7c9-43c3-b136-6cacb5137565" containerID="75949f44bdd997a0febe659a765990c426577bbd1979336a01af5aa0437ba26e" exitCode=0
Oct 30 20:12:50.324413 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.324320    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"
Oct 30 20:12:50.324413 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.324333    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w" event={"ID":"ea94ede7-d7c9-43c3-b136-6cacb5137565","Type":"ContainerDied","Data":"75949f44bdd997a0febe659a765990c426577bbd1979336a01af5aa0437ba26e"}
Oct 30 20:12:50.324413 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.324385    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w" event={"ID":"ea94ede7-d7c9-43c3-b136-6cacb5137565","Type":"ContainerDied","Data":"70f2bf41c86809a219b67b16fab781223144c2559c9ab42f8d4cd23e555f7bf2"}
Oct 30 20:12:50.324413 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.324406    6030 scope.go:122] "RemoveContainer" containerID="75949f44bdd997a0febe659a765990c426577bbd1979336a01af5aa0437ba26e"
Oct 30 20:12:50.377481 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.377457    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"]
Oct 30 20:12:50.386320 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.386291    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645vzl9w"]
Oct 30 20:12:50.386576 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.386556    6030 scope.go:122] "RemoveContainer" containerID="75949f44bdd997a0febe659a765990c426577bbd1979336a01af5aa0437ba26e"
Oct 30 20:12:50.386935 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:12:50.386909    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75949f44bdd997a0febe659a765990c426577bbd1979336a01af5aa0437ba26e\": container with ID starting with 75949f44bdd997a0febe659a765990c426577bbd1979336a01af5aa0437ba26e not found: ID does not exist" containerID="75949f44bdd997a0febe659a765990c426577bbd1979336a01af5aa0437ba26e"
Oct 30 20:12:50.387025 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.386949    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75949f44bdd997a0febe659a765990c426577bbd1979336a01af5aa0437ba26e"} err="failed to get container status \"75949f44bdd997a0febe659a765990c426577bbd1979336a01af5aa0437ba26e\": rpc error: code = NotFound desc = could not find container \"75949f44bdd997a0febe659a765990c426577bbd1979336a01af5aa0437ba26e\": container with ID starting with 75949f44bdd997a0febe659a765990c426577bbd1979336a01af5aa0437ba26e not found: ID does not exist"
Oct 30 20:12:50.565510 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.565483    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:50.678975 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.678934    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/220d793a-4370-4d29-a3d1-7fe3903afca8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/220d793a-4370-4d29-a3d1-7fe3903afca8-plugins-config-volume\") pod \"220d793a-4370-4d29-a3d1-7fe3903afca8\" (UID: \"220d793a-4370-4d29-a3d1-7fe3903afca8\") "
Oct 30 20:12:50.679194 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.679162    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/220d793a-4370-4d29-a3d1-7fe3903afca8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/220d793a-4370-4d29-a3d1-7fe3903afca8-routing-overlay-volume\") pod \"220d793a-4370-4d29-a3d1-7fe3903afca8\" (UID: \"220d793a-4370-4d29-a3d1-7fe3903afca8\") "
Oct 30 20:12:50.679261 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.679203    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220d793a-4370-4d29-a3d1-7fe3903afca8-kube-api-access-tvgr5\" (UniqueName: \"kubernetes.io/projected/220d793a-4370-4d29-a3d1-7fe3903afca8-kube-api-access-tvgr5\") pod \"220d793a-4370-4d29-a3d1-7fe3903afca8\" (UID: \"220d793a-4370-4d29-a3d1-7fe3903afca8\") "
Oct 30 20:12:50.679354 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.679333    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220d793a-4370-4d29-a3d1-7fe3903afca8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/220d793a-4370-4d29-a3d1-7fe3903afca8-provider-credentials\") pod \"220d793a-4370-4d29-a3d1-7fe3903afca8\" (UID: \"220d793a-4370-4d29-a3d1-7fe3903afca8\") "
Oct 30 20:12:50.681426 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.680480    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220d793a-4370-4d29-a3d1-7fe3903afca8-plugins-config-volume" pod "220d793a-4370-4d29-a3d1-7fe3903afca8" (UID: "220d793a-4370-4d29-a3d1-7fe3903afca8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:50.681840 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.681762    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220d793a-4370-4d29-a3d1-7fe3903afca8-routing-overlay-volume" pod "220d793a-4370-4d29-a3d1-7fe3903afca8" (UID: "220d793a-4370-4d29-a3d1-7fe3903afca8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:50.698974 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.698939    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220d793a-4370-4d29-a3d1-7fe3903afca8-kube-api-access-tvgr5" pod "220d793a-4370-4d29-a3d1-7fe3903afca8" (UID: "220d793a-4370-4d29-a3d1-7fe3903afca8"). InnerVolumeSpecName "kube-api-access-tvgr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:50.704026 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.700069    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220d793a-4370-4d29-a3d1-7fe3903afca8-provider-credentials" pod "220d793a-4370-4d29-a3d1-7fe3903afca8" (UID: "220d793a-4370-4d29-a3d1-7fe3903afca8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:50.781509 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.781432    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/220d793a-4370-4d29-a3d1-7fe3903afca8-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.781509 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.781464    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/220d793a-4370-4d29-a3d1-7fe3903afca8-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.781509 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.781475    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/220d793a-4370-4d29-a3d1-7fe3903afca8-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.781509 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:50.781483    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvgr5\" (UniqueName: \"kubernetes.io/projected/220d793a-4370-4d29-a3d1-7fe3903afca8-kube-api-access-tvgr5\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:51.203093 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:51.203038    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea94ede7-d7c9-43c3-b136-6cacb5137565" path="/var/lib/kubelet/pods/ea94ede7-d7c9-43c3-b136-6cacb5137565/volumes"
Oct 30 20:12:51.333771 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:51.333728    6030 generic.go:356] "Generic (PLEG): container finished" podID="220d793a-4370-4d29-a3d1-7fe3903afca8" containerID="6142b726d71e8e33d8e5a2a020f77f1538a3fbc769439321aed66f8da17f80cf" exitCode=0
Oct 30 20:12:51.334302 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:51.333803    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"
Oct 30 20:12:51.334302 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:51.333803    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6" event={"ID":"220d793a-4370-4d29-a3d1-7fe3903afca8","Type":"ContainerDied","Data":"6142b726d71e8e33d8e5a2a020f77f1538a3fbc769439321aed66f8da17f80cf"}
Oct 30 20:12:51.334302 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:51.333928    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6" event={"ID":"220d793a-4370-4d29-a3d1-7fe3903afca8","Type":"ContainerDied","Data":"a588e53d3fcf4022c7e926de36ff857ef7985153b79bbff0196510e8912471c9"}
Oct 30 20:12:51.334302 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:51.333951    6030 scope.go:122] "RemoveContainer" containerID="6142b726d71e8e33d8e5a2a020f77f1538a3fbc769439321aed66f8da17f80cf"
Oct 30 20:12:51.344437 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:51.344420    6030 scope.go:122] "RemoveContainer" containerID="6142b726d71e8e33d8e5a2a020f77f1538a3fbc769439321aed66f8da17f80cf"
Oct 30 20:12:51.344709 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:12:51.344690    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6142b726d71e8e33d8e5a2a020f77f1538a3fbc769439321aed66f8da17f80cf\": container with ID starting with 6142b726d71e8e33d8e5a2a020f77f1538a3fbc769439321aed66f8da17f80cf not found: ID does not exist" containerID="6142b726d71e8e33d8e5a2a020f77f1538a3fbc769439321aed66f8da17f80cf"
Oct 30 20:12:51.344765 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:51.344719    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6142b726d71e8e33d8e5a2a020f77f1538a3fbc769439321aed66f8da17f80cf"} err="failed to get container status \"6142b726d71e8e33d8e5a2a020f77f1538a3fbc769439321aed66f8da17f80cf\": rpc error: code = NotFound desc = could not find container \"6142b726d71e8e33d8e5a2a020f77f1538a3fbc769439321aed66f8da17f80cf\": container with ID starting with 6142b726d71e8e33d8e5a2a020f77f1538a3fbc769439321aed66f8da17f80cf not found: ID does not exist"
Oct 30 20:12:51.356574 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:51.356553    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"]
Oct 30 20:12:51.361699 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:51.361679    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbc4vjl6"]
Oct 30 20:12:53.204795 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:12:53.204764    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="220d793a-4370-4d29-a3d1-7fe3903afca8" path="/var/lib/kubelet/pods/220d793a-4370-4d29-a3d1-7fe3903afca8/volumes"
Oct 30 20:13:04.290792 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:13:04.290740    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:13:10.667474 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.667433    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"]
Oct 30 20:13:10.669285 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.669252    6030 cpu_manager.go:414] "containerMap: removing container" podUID="ea94ede7-d7c9-43c3-b136-6cacb5137565" containerName="payload-processing"
Oct 30 20:13:10.669285 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.669283    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea94ede7-d7c9-43c3-b136-6cacb5137565" containerName="payload-processing"
Oct 30 20:13:10.669511 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.669320    6030 cpu_manager.go:414] "containerMap: removing container" podUID="220d793a-4370-4d29-a3d1-7fe3903afca8" containerName="payload-processing"
Oct 30 20:13:10.669511 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.669329    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="220d793a-4370-4d29-a3d1-7fe3903afca8" containerName="payload-processing"
Oct 30 20:13:10.669511 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.669502    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea94ede7-d7c9-43c3-b136-6cacb5137565" containerName="payload-processing"
Oct 30 20:13:10.669678 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.669520    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="220d793a-4370-4d29-a3d1-7fe3903afca8" containerName="payload-processing"
Oct 30 20:13:10.674170 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.674144    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:13:10.679218 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.679190    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"payload-processing-external-model-e2e--632181a7-dockercfg-dd2cn\""
Oct 30 20:13:10.680718 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.680693    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"agc-cred-4a17ee2c812f\""
Oct 30 20:13:10.680882 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.680713    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"agc-cred-e787e8649e56\""
Oct 30 20:13:10.681475 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.681456    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4d5758\""
Oct 30 20:13:10.681597 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.681555    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"kube-root-ca.crt\""
Oct 30 20:13:10.681735 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.681583    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"openshift-service-ca.crt\""
Oct 30 20:13:10.682621 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.682586    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"routing-overlay\""
Oct 30 20:13:10.692637 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.692601    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"]
Oct 30 20:13:10.751815 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.751777    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj\" (UID: \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:13:10.751995 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.751913    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggb28\" (UniqueName: \"kubernetes.io/projected/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-kube-api-access-ggb28\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj\" (UID: \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:13:10.752101 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.751998    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj\" (UID: \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:13:10.752101 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.752071    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj\" (UID: \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:13:10.853299 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.853263    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj\" (UID: \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:13:10.853483 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.853360    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggb28\" (UniqueName: \"kubernetes.io/projected/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-kube-api-access-ggb28\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj\" (UID: \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:13:10.853483 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.853462    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj\" (UID: \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:13:10.853600 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.853499    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj\" (UID: \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:13:10.854136 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.854098    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj\" (UID: \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:13:10.854454 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.854143    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj\" (UID: \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:13:10.856233 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.856214    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj\" (UID: \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:13:10.868722 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.868701    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggb28\" (UniqueName: \"kubernetes.io/projected/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-kube-api-access-ggb28\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj\" (UID: \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:13:10.994978 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:10.994905    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:13:11.104363 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.104328    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"]
Oct 30 20:13:11.109220 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.109188    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:13:11.118883 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.118807    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"agc-cred-c8bb93f01837\""
Oct 30 20:13:11.119032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.118953    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"routing-overlay\""
Oct 30 20:13:11.119032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.118990    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"agc-cred-7233bc96d635\""
Oct 30 20:13:11.120420 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.119783    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"kube-root-ca.crt\""
Oct 30 20:13:11.120420 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.119933    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"payload-processing-external-model-plugins-e2e-extmod-a-222480\""
Oct 30 20:13:11.120420 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.120001    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"payload-processing-external-model-e2e--8e4491f3-dockercfg-rdtdr\""
Oct 30 20:13:11.120420 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.120126    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"openshift-service-ca.crt\""
Oct 30 20:13:11.123134 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.123113    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"]
Oct 30 20:13:11.259417 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.259324    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80c90f1-2746-4a9b-a821-5c94a5cc8241-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h\" (UID: \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:13:11.259660 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.259634    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a80c90f1-2746-4a9b-a821-5c94a5cc8241-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h\" (UID: \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:13:11.259747 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.259681    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a80c90f1-2746-4a9b-a821-5c94a5cc8241-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h\" (UID: \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:13:11.259809 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.259791    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnvsj\" (UniqueName: \"kubernetes.io/projected/a80c90f1-2746-4a9b-a821-5c94a5cc8241-kube-api-access-bnvsj\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h\" (UID: \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:13:11.285639 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.285604    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"]
Oct 30 20:13:11.286743 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:13:11.286717    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d09bca_f401_48bf_b2c2_8395bd8c2cf3.slice/crio-17565f3c94829b45d24096f6c0c41f4fda90d5dec7c3903c7e496544b5440147 WatchSource:0}: Error finding container 17565f3c94829b45d24096f6c0c41f4fda90d5dec7c3903c7e496544b5440147: Status 404 returned error can't find the container with id 17565f3c94829b45d24096f6c0c41f4fda90d5dec7c3903c7e496544b5440147
Oct 30 20:13:11.361400 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.361362    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80c90f1-2746-4a9b-a821-5c94a5cc8241-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h\" (UID: \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:13:11.361540 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.361511    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a80c90f1-2746-4a9b-a821-5c94a5cc8241-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h\" (UID: \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:13:11.361540 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.361538    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a80c90f1-2746-4a9b-a821-5c94a5cc8241-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h\" (UID: \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:13:11.361659 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.361569    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvsj\" (UniqueName: \"kubernetes.io/projected/a80c90f1-2746-4a9b-a821-5c94a5cc8241-kube-api-access-bnvsj\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h\" (UID: \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:13:11.362164 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.362131    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a80c90f1-2746-4a9b-a821-5c94a5cc8241-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h\" (UID: \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:13:11.362269 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.362232    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80c90f1-2746-4a9b-a821-5c94a5cc8241-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h\" (UID: \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:13:11.364087 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.364068    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a80c90f1-2746-4a9b-a821-5c94a5cc8241-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h\" (UID: \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:13:11.380872 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.380849    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnvsj\" (UniqueName: \"kubernetes.io/projected/a80c90f1-2746-4a9b-a821-5c94a5cc8241-kube-api-access-bnvsj\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h\" (UID: \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:13:11.434160 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.434132    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:13:11.489841 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.489687    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj" event={"ID":"49d09bca-f401-48bf-b2c2-8395bd8c2cf3","Type":"ContainerStarted","Data":"4ac7b4f21a2d3db4f8a08d913498d2942e0fee3e40ecc6f196748eebe25dc666"}
Oct 30 20:13:11.489841 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.489754    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj" event={"ID":"49d09bca-f401-48bf-b2c2-8395bd8c2cf3","Type":"ContainerStarted","Data":"17565f3c94829b45d24096f6c0c41f4fda90d5dec7c3903c7e496544b5440147"}
Oct 30 20:13:11.489841 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.489808    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:13:11.513741 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.513651    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj" podStartSLOduration=1.513637434 podStartE2EDuration="1.513637434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:11.511327443 +0000 UTC m=+2470.426934300" watchObservedRunningTime="2026-10-30 20:13:11.513637434 +0000 UTC m=+2470.429244306"
Oct 30 20:13:11.722522 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:11.722494    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"]
Oct 30 20:13:11.722873 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:13:11.722651    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80c90f1_2746_4a9b_a821_5c94a5cc8241.slice/crio-90c84e32c7c526f1537f34e1787bb70ba626cc8ebbff4e555e0812a4ee9da282 WatchSource:0}: Error finding container 90c84e32c7c526f1537f34e1787bb70ba626cc8ebbff4e555e0812a4ee9da282: Status 404 returned error can't find the container with id 90c84e32c7c526f1537f34e1787bb70ba626cc8ebbff4e555e0812a4ee9da282
Oct 30 20:13:12.499199 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:12.499163    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h" event={"ID":"a80c90f1-2746-4a9b-a821-5c94a5cc8241","Type":"ContainerStarted","Data":"98322070e921a20f398cd6e5aa4bf90cb1ae0a14861cc52b9846165a92a473e6"}
Oct 30 20:13:12.499425 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:12.499217    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h" event={"ID":"a80c90f1-2746-4a9b-a821-5c94a5cc8241","Type":"ContainerStarted","Data":"90c84e32c7c526f1537f34e1787bb70ba626cc8ebbff4e555e0812a4ee9da282"}
Oct 30 20:13:12.548400 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:12.548344    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h" podStartSLOduration=1.548325189 podStartE2EDuration="1.548325189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:12.54464815 +0000 UTC m=+2471.460255006" watchObservedRunningTime="2026-10-30 20:13:12.548325189 +0000 UTC m=+2471.463932045"
Oct 30 20:13:13.506295 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:13.506264    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:13:22.502263 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:22.502234    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:13:24.516732 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:13:24.516691    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:13:34.290175 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:13:34.290138    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:14:04.297903 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:14:04.297864    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:14:06.118428 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.118391    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"]
Oct 30 20:14:06.123255 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.123233    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:06.134536 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.133260    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"agc-cred-20d4e7c9cc7c\""
Oct 30 20:14:06.138066 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.137373    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"]
Oct 30 20:14:06.239586 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.239550    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl\" (UID: \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:06.239770 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.239628    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5km4\" (UniqueName: \"kubernetes.io/projected/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-kube-api-access-k5km4\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl\" (UID: \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:06.239825 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.239785    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl\" (UID: \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:06.239976 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.239951    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl\" (UID: \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:06.341797 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.341754    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl\" (UID: \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:06.341993 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.341879    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl\" (UID: \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:06.341993 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.341939    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl\" (UID: \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:06.342182 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.342010    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5km4\" (UniqueName: \"kubernetes.io/projected/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-kube-api-access-k5km4\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl\" (UID: \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:06.342552 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.342525    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl\" (UID: \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:06.342688 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.342667    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl\" (UID: \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:06.344754 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.344734    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl\" (UID: \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:06.358018 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.357995    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5km4\" (UniqueName: \"kubernetes.io/projected/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-kube-api-access-k5km4\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl\" (UID: \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:06.447380 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.447352    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:06.511256 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.511200    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"]
Oct 30 20:14:06.516467 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.516435    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:06.527223 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.524365    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"agc-cred-bd77348f7e09\""
Oct 30 20:14:06.554258 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.543083    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"]
Oct 30 20:14:06.646397 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.646364    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24e84582-fba1-41f5-b4e1-bd19d13f33db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz\" (UID: \"24e84582-fba1-41f5-b4e1-bd19d13f33db\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:06.646560 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.646406    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24e84582-fba1-41f5-b4e1-bd19d13f33db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz\" (UID: \"24e84582-fba1-41f5-b4e1-bd19d13f33db\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:06.646560 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.646536    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2dd6\" (UniqueName: \"kubernetes.io/projected/24e84582-fba1-41f5-b4e1-bd19d13f33db-kube-api-access-f2dd6\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz\" (UID: \"24e84582-fba1-41f5-b4e1-bd19d13f33db\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:06.646662 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.646640    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e84582-fba1-41f5-b4e1-bd19d13f33db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz\" (UID: \"24e84582-fba1-41f5-b4e1-bd19d13f33db\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:06.748891 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.748581    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24e84582-fba1-41f5-b4e1-bd19d13f33db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz\" (UID: \"24e84582-fba1-41f5-b4e1-bd19d13f33db\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:06.749192 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.749163    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24e84582-fba1-41f5-b4e1-bd19d13f33db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz\" (UID: \"24e84582-fba1-41f5-b4e1-bd19d13f33db\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:06.749378 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.749284    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2dd6\" (UniqueName: \"kubernetes.io/projected/24e84582-fba1-41f5-b4e1-bd19d13f33db-kube-api-access-f2dd6\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz\" (UID: \"24e84582-fba1-41f5-b4e1-bd19d13f33db\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:06.749454 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.749378    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e84582-fba1-41f5-b4e1-bd19d13f33db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz\" (UID: \"24e84582-fba1-41f5-b4e1-bd19d13f33db\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:06.750232 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.750208    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e84582-fba1-41f5-b4e1-bd19d13f33db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz\" (UID: \"24e84582-fba1-41f5-b4e1-bd19d13f33db\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:06.750372 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.750343    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24e84582-fba1-41f5-b4e1-bd19d13f33db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz\" (UID: \"24e84582-fba1-41f5-b4e1-bd19d13f33db\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:06.752356 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.752202    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24e84582-fba1-41f5-b4e1-bd19d13f33db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz\" (UID: \"24e84582-fba1-41f5-b4e1-bd19d13f33db\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:06.753750 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.753720    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"]
Oct 30 20:14:06.759383 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:14:06.759355    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd426ce34_85fa_4a6f_8d0e_dfe7ead02afb.slice/crio-6079fbcececdfa9e6ffe89fcc45a8c868abe2a2530ef79cfbc940a9f8a15fcc4 WatchSource:0}: Error finding container 6079fbcececdfa9e6ffe89fcc45a8c868abe2a2530ef79cfbc940a9f8a15fcc4: Status 404 returned error can't find the container with id 6079fbcececdfa9e6ffe89fcc45a8c868abe2a2530ef79cfbc940a9f8a15fcc4
Oct 30 20:14:06.765418 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.765331    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2dd6\" (UniqueName: \"kubernetes.io/projected/24e84582-fba1-41f5-b4e1-bd19d13f33db-kube-api-access-f2dd6\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz\" (UID: \"24e84582-fba1-41f5-b4e1-bd19d13f33db\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:06.848734 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.848703    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:06.912289 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.912249    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl" event={"ID":"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb","Type":"ContainerStarted","Data":"33efe897d29d2f4a2e5d9bd7773a1bb43fcc62663850654f8a123e6c4719bc38"}
Oct 30 20:14:06.912406 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.912315    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl" event={"ID":"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb","Type":"ContainerStarted","Data":"6079fbcececdfa9e6ffe89fcc45a8c868abe2a2530ef79cfbc940a9f8a15fcc4"}
Oct 30 20:14:06.912406 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.912343    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:06.939506 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:06.939456    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl" podStartSLOduration=0.939442667 podStartE2EDuration="939.442667ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:06.937733764 +0000 UTC m=+2525.853340630" watchObservedRunningTime="2026-10-30 20:14:06.939442667 +0000 UTC m=+2525.855049522"
Oct 30 20:14:07.135711 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:14:07.135678    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e84582_fba1_41f5_b4e1_bd19d13f33db.slice/crio-16c61d01414f25c921ca73bf4be720edd36234ef8f4fd1bedfbd966f430c49cb WatchSource:0}: Error finding container 16c61d01414f25c921ca73bf4be720edd36234ef8f4fd1bedfbd966f430c49cb: Status 404 returned error can't find the container with id 16c61d01414f25c921ca73bf4be720edd36234ef8f4fd1bedfbd966f430c49cb
Oct 30 20:14:07.136240 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:07.136081    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"]
Oct 30 20:14:07.921008 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:07.920969    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz" event={"ID":"24e84582-fba1-41f5-b4e1-bd19d13f33db","Type":"ContainerStarted","Data":"8cedd609f1e98381906cbcb65d7b8171aea0784d33ed6265c7997eda1f156629"}
Oct 30 20:14:07.921252 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:07.921024    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz" event={"ID":"24e84582-fba1-41f5-b4e1-bd19d13f33db","Type":"ContainerStarted","Data":"16c61d01414f25c921ca73bf4be720edd36234ef8f4fd1bedfbd966f430c49cb"}
Oct 30 20:14:07.921252 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:07.921209    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:07.946567 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:07.946515    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz" podStartSLOduration=1.946473938 podStartE2EDuration="1.946473938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:07.94526387 +0000 UTC m=+2526.860870726" watchObservedRunningTime="2026-10-30 20:14:07.946473938 +0000 UTC m=+2526.862080795"
Oct 30 20:14:12.570811 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:12.570770    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"]
Oct 30 20:14:12.576599 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:12.576571    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"
Oct 30 20:14:12.580129 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:12.580103    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"]
Oct 30 20:14:12.582321 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:12.582294    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-a41ade\"/\"kube-root-ca.crt\""
Oct 30 20:14:12.582500 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:12.582475    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-a41ade\"/\"openshift-service-ca.crt\""
Oct 30 20:14:12.584015 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:12.583877    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-a41ade\"/\"e2e-usage-sim-1b1eae-cert\""
Oct 30 20:14:12.584015 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:12.583917    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-a41ade\"/\"default-dockercfg-ptbb9\""
Oct 30 20:14:12.731732 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:12.731700    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b259c906-057d-47dd-ace7-630c8ade49fb-tls\") pod \"e2e-usage-sim-1b1eae-774689b76c-fljn8\" (UID: \"b259c906-057d-47dd-ace7-630c8ade49fb\") " pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"
Oct 30 20:14:12.731986 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:12.731966    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br7gv\" (UniqueName: \"kubernetes.io/projected/b259c906-057d-47dd-ace7-630c8ade49fb-kube-api-access-br7gv\") pod \"e2e-usage-sim-1b1eae-774689b76c-fljn8\" (UID: \"b259c906-057d-47dd-ace7-630c8ade49fb\") " pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"
Oct 30 20:14:12.832976 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:12.832891    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br7gv\" (UniqueName: \"kubernetes.io/projected/b259c906-057d-47dd-ace7-630c8ade49fb-kube-api-access-br7gv\") pod \"e2e-usage-sim-1b1eae-774689b76c-fljn8\" (UID: \"b259c906-057d-47dd-ace7-630c8ade49fb\") " pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"
Oct 30 20:14:12.833193 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:12.833013    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b259c906-057d-47dd-ace7-630c8ade49fb-tls\") pod \"e2e-usage-sim-1b1eae-774689b76c-fljn8\" (UID: \"b259c906-057d-47dd-ace7-630c8ade49fb\") " pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"
Oct 30 20:14:12.835707 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:12.835681    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b259c906-057d-47dd-ace7-630c8ade49fb-tls\") pod \"e2e-usage-sim-1b1eae-774689b76c-fljn8\" (UID: \"b259c906-057d-47dd-ace7-630c8ade49fb\") " pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"
Oct 30 20:14:12.846878 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:12.846854    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br7gv\" (UniqueName: \"kubernetes.io/projected/b259c906-057d-47dd-ace7-630c8ade49fb-kube-api-access-br7gv\") pod \"e2e-usage-sim-1b1eae-774689b76c-fljn8\" (UID: \"b259c906-057d-47dd-ace7-630c8ade49fb\") " pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"
Oct 30 20:14:12.894846 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:12.894822    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"
Oct 30 20:14:13.176641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:13.176615    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"]
Oct 30 20:14:13.177169 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:14:13.177137    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb259c906_057d_47dd_ace7_630c8ade49fb.slice/crio-cfa757a153acd71443c00f589ae046635ac2260ebc90aba289482677117362c9 WatchSource:0}: Error finding container cfa757a153acd71443c00f589ae046635ac2260ebc90aba289482677117362c9: Status 404 returned error can't find the container with id cfa757a153acd71443c00f589ae046635ac2260ebc90aba289482677117362c9
Oct 30 20:14:13.976866 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:13.976826    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8" event={"ID":"b259c906-057d-47dd-ace7-630c8ade49fb","Type":"ContainerStarted","Data":"f223640f6ecceca05121631dc89de2372712d3d75366ea09431c1ad79a186a48"}
Oct 30 20:14:13.977456 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:13.976879    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8" event={"ID":"b259c906-057d-47dd-ace7-630c8ade49fb","Type":"ContainerStarted","Data":"cfa757a153acd71443c00f589ae046635ac2260ebc90aba289482677117362c9"}
Oct 30 20:14:13.977456 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:13.977215    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"
Oct 30 20:14:13.983006 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:13.982975    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"
Oct 30 20:14:14.000676 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:14.000617    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8" podStartSLOduration=2.000578806 podStartE2EDuration="2.000578806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:13.996499175 +0000 UTC m=+2532.912106033" watchObservedRunningTime="2026-10-30 20:14:14.000578806 +0000 UTC m=+2532.916185678"
Oct 30 20:14:15.712353 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:15.712319    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"]
Oct 30 20:14:15.712862 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:15.712511    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl" podUID="039cd584-b9e1-4b28-b0d5-5c482cbdaad3" containerName="istio-proxy" containerID="cri-o://a046b9f002e980c6e654ea3b39fcc95a3e02a1439f48988672ff9bba5879dfbd" gracePeriod=30
Oct 30 20:14:17.925520 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:17.925486    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:18.009786 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:18.009752    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"]
Oct 30 20:14:18.945072 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:18.943656    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:19.058069 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:19.057884    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"]
Oct 30 20:14:20.986739 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:20.986677    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 20:14:21.081391 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.081355    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-token\" (UniqueName: \"kubernetes.io/projected/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-token\") pod \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") "
Oct 30 20:14:21.081595 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.081430    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-kube-api-access-qvpvs\" (UniqueName: \"kubernetes.io/projected/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-kube-api-access-qvpvs\") pod \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") "
Oct 30 20:14:21.081595 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.081464    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-credential-socket\") pod \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") "
Oct 30 20:14:21.081595 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.081492    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istiod-ca-cert\") pod \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") "
Oct 30 20:14:21.081780 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.081633    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-workload-certs\") pod \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") "
Oct 30 20:14:21.081848 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.081799    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-credential-socket" pod "039cd584-b9e1-4b28-b0d5-5c482cbdaad3" (UID: "039cd584-b9e1-4b28-b0d5-5c482cbdaad3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:21.081907 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.081847    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-podinfo\") pod \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") "
Oct 30 20:14:21.081961 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.081894    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-workload-certs" pod "039cd584-b9e1-4b28-b0d5-5c482cbdaad3" (UID: "039cd584-b9e1-4b28-b0d5-5c482cbdaad3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:21.081961 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.081924    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-workload-socket\") pod \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") "
Oct 30 20:14:21.082105 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.081964    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-data\") pod \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") "
Oct 30 20:14:21.082105 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.082012    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-envoy\") pod \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\" (UID: \"039cd584-b9e1-4b28-b0d5-5c482cbdaad3\") "
Oct 30 20:14:21.082323 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.082298    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-data" pod "039cd584-b9e1-4b28-b0d5-5c482cbdaad3" (UID: "039cd584-b9e1-4b28-b0d5-5c482cbdaad3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:21.082391 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.082373    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-workload-socket" pod "039cd584-b9e1-4b28-b0d5-5c482cbdaad3" (UID: "039cd584-b9e1-4b28-b0d5-5c482cbdaad3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:21.083079 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.083012    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istiod-ca-cert" pod "039cd584-b9e1-4b28-b0d5-5c482cbdaad3" (UID: "039cd584-b9e1-4b28-b0d5-5c482cbdaad3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:21.083548 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.083525    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:21.083664 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.083553    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:21.083664 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.083570    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:21.083664 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.083584    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:21.083664 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.083600    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:21.084348 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.084319    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-kube-api-access-qvpvs" pod "039cd584-b9e1-4b28-b0d5-5c482cbdaad3" (UID: "039cd584-b9e1-4b28-b0d5-5c482cbdaad3"). InnerVolumeSpecName "kube-api-access-qvpvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:21.084471 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.084446    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-token" pod "039cd584-b9e1-4b28-b0d5-5c482cbdaad3" (UID: "039cd584-b9e1-4b28-b0d5-5c482cbdaad3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:21.084579 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.084560    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-envoy" pod "039cd584-b9e1-4b28-b0d5-5c482cbdaad3" (UID: "039cd584-b9e1-4b28-b0d5-5c482cbdaad3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:21.084791 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.084762    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-podinfo" pod "039cd584-b9e1-4b28-b0d5-5c482cbdaad3" (UID: "039cd584-b9e1-4b28-b0d5-5c482cbdaad3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:14:21.087201 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.087176    6030 generic.go:356] "Generic (PLEG): container finished" podID="039cd584-b9e1-4b28-b0d5-5c482cbdaad3" containerID="a046b9f002e980c6e654ea3b39fcc95a3e02a1439f48988672ff9bba5879dfbd" exitCode=0
Oct 30 20:14:21.087300 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.087234    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"
Oct 30 20:14:21.087300 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.087262    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl" event={"ID":"039cd584-b9e1-4b28-b0d5-5c482cbdaad3","Type":"ContainerDied","Data":"a046b9f002e980c6e654ea3b39fcc95a3e02a1439f48988672ff9bba5879dfbd"}
Oct 30 20:14:21.087388 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.087358    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl" event={"ID":"039cd584-b9e1-4b28-b0d5-5c482cbdaad3","Type":"ContainerDied","Data":"1375c84e0a4078e436602b76a30dd7c35d3f344ea5828a0097ae3fade5ebf43b"}
Oct 30 20:14:21.087428 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.087389    6030 scope.go:122] "RemoveContainer" containerID="a046b9f002e980c6e654ea3b39fcc95a3e02a1439f48988672ff9bba5879dfbd"
Oct 30 20:14:21.116644 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.116625    6030 scope.go:122] "RemoveContainer" containerID="a046b9f002e980c6e654ea3b39fcc95a3e02a1439f48988672ff9bba5879dfbd"
Oct 30 20:14:21.116970 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:14:21.116950    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a046b9f002e980c6e654ea3b39fcc95a3e02a1439f48988672ff9bba5879dfbd\": container with ID starting with a046b9f002e980c6e654ea3b39fcc95a3e02a1439f48988672ff9bba5879dfbd not found: ID does not exist" containerID="a046b9f002e980c6e654ea3b39fcc95a3e02a1439f48988672ff9bba5879dfbd"
Oct 30 20:14:21.117038 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.116979    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a046b9f002e980c6e654ea3b39fcc95a3e02a1439f48988672ff9bba5879dfbd"} err="failed to get container status \"a046b9f002e980c6e654ea3b39fcc95a3e02a1439f48988672ff9bba5879dfbd\": rpc error: code = NotFound desc = could not find container \"a046b9f002e980c6e654ea3b39fcc95a3e02a1439f48988672ff9bba5879dfbd\": container with ID starting with a046b9f002e980c6e654ea3b39fcc95a3e02a1439f48988672ff9bba5879dfbd not found: ID does not exist"
Oct 30 20:14:21.131556 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.131527    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"]
Oct 30 20:14:21.135822 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.135802    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-7wvdl"]
Oct 30 20:14:21.184995 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.184957    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:21.184995 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.184987    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:21.184995 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.184998    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:21.185321 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.185007    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvpvs\" (UniqueName: \"kubernetes.io/projected/039cd584-b9e1-4b28-b0d5-5c482cbdaad3-kube-api-access-qvpvs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:21.212737 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:21.212707    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="039cd584-b9e1-4b28-b0d5-5c482cbdaad3" path="/var/lib/kubelet/pods/039cd584-b9e1-4b28-b0d5-5c482cbdaad3/volumes"
Oct 30 20:14:23.010107 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:23.010071    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj" podUID="49d09bca-f401-48bf-b2c2-8395bd8c2cf3" containerName="payload-processing" containerID="cri-o://4ac7b4f21a2d3db4f8a08d913498d2942e0fee3e40ecc6f196748eebe25dc666" gracePeriod=25
Oct 30 20:14:23.294349 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:23.294323    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:14:23.417673 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:23.417645    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-routing-overlay-volume\") pod \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\" (UID: \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\") "
Oct 30 20:14:23.417867 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:23.417774    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-plugins-config-volume\") pod \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\" (UID: \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\") "
Oct 30 20:14:23.417867 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:23.417826    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-provider-credentials\") pod \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\" (UID: \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\") "
Oct 30 20:14:23.417867 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:23.417866    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-kube-api-access-ggb28\" (UniqueName: \"kubernetes.io/projected/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-kube-api-access-ggb28\") pod \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\" (UID: \"49d09bca-f401-48bf-b2c2-8395bd8c2cf3\") "
Oct 30 20:14:23.418190 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:23.418161    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-routing-overlay-volume" pod "49d09bca-f401-48bf-b2c2-8395bd8c2cf3" (UID: "49d09bca-f401-48bf-b2c2-8395bd8c2cf3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:23.418317 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:23.418295    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-plugins-config-volume" pod "49d09bca-f401-48bf-b2c2-8395bd8c2cf3" (UID: "49d09bca-f401-48bf-b2c2-8395bd8c2cf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:23.418600 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:23.418540    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:23.418600 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:23.418561    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:23.420390 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:23.420364    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-kube-api-access-ggb28" pod "49d09bca-f401-48bf-b2c2-8395bd8c2cf3" (UID: "49d09bca-f401-48bf-b2c2-8395bd8c2cf3"). InnerVolumeSpecName "kube-api-access-ggb28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:23.420490 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:23.420475    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-provider-credentials" pod "49d09bca-f401-48bf-b2c2-8395bd8c2cf3" (UID: "49d09bca-f401-48bf-b2c2-8395bd8c2cf3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:23.519949 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:23.519868    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:23.519949 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:23.519900    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggb28\" (UniqueName: \"kubernetes.io/projected/49d09bca-f401-48bf-b2c2-8395bd8c2cf3-kube-api-access-ggb28\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:24.059122 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.059029    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h" podUID="a80c90f1-2746-4a9b-a821-5c94a5cc8241" containerName="payload-processing" containerID="cri-o://98322070e921a20f398cd6e5aa4bf90cb1ae0a14861cc52b9846165a92a473e6" gracePeriod=25
Oct 30 20:14:24.117150 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.117114    6030 generic.go:356] "Generic (PLEG): container finished" podID="49d09bca-f401-48bf-b2c2-8395bd8c2cf3" containerID="4ac7b4f21a2d3db4f8a08d913498d2942e0fee3e40ecc6f196748eebe25dc666" exitCode=0
Oct 30 20:14:24.117347 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.117181    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"
Oct 30 20:14:24.117347 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.117202    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj" event={"ID":"49d09bca-f401-48bf-b2c2-8395bd8c2cf3","Type":"ContainerDied","Data":"4ac7b4f21a2d3db4f8a08d913498d2942e0fee3e40ecc6f196748eebe25dc666"}
Oct 30 20:14:24.117347 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.117272    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj" event={"ID":"49d09bca-f401-48bf-b2c2-8395bd8c2cf3","Type":"ContainerDied","Data":"17565f3c94829b45d24096f6c0c41f4fda90d5dec7c3903c7e496544b5440147"}
Oct 30 20:14:24.117347 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.117306    6030 scope.go:122] "RemoveContainer" containerID="4ac7b4f21a2d3db4f8a08d913498d2942e0fee3e40ecc6f196748eebe25dc666"
Oct 30 20:14:24.137242 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.137221    6030 scope.go:122] "RemoveContainer" containerID="4ac7b4f21a2d3db4f8a08d913498d2942e0fee3e40ecc6f196748eebe25dc666"
Oct 30 20:14:24.137681 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:14:24.137649    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac7b4f21a2d3db4f8a08d913498d2942e0fee3e40ecc6f196748eebe25dc666\": container with ID starting with 4ac7b4f21a2d3db4f8a08d913498d2942e0fee3e40ecc6f196748eebe25dc666 not found: ID does not exist" containerID="4ac7b4f21a2d3db4f8a08d913498d2942e0fee3e40ecc6f196748eebe25dc666"
Oct 30 20:14:24.137807 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.137687    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac7b4f21a2d3db4f8a08d913498d2942e0fee3e40ecc6f196748eebe25dc666"} err="failed to get container status \"4ac7b4f21a2d3db4f8a08d913498d2942e0fee3e40ecc6f196748eebe25dc666\": rpc error: code = NotFound desc = could not find container \"4ac7b4f21a2d3db4f8a08d913498d2942e0fee3e40ecc6f196748eebe25dc666\": container with ID starting with 4ac7b4f21a2d3db4f8a08d913498d2942e0fee3e40ecc6f196748eebe25dc666 not found: ID does not exist"
Oct 30 20:14:24.159983 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.159955    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"]
Oct 30 20:14:24.165747 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.165724    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f82h5sj"]
Oct 30 20:14:24.442763 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.442728    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:14:24.532996 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.532963    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a80c90f1-2746-4a9b-a821-5c94a5cc8241-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80c90f1-2746-4a9b-a821-5c94a5cc8241-plugins-config-volume\") pod \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\" (UID: \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\") "
Oct 30 20:14:24.533196 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.533007    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a80c90f1-2746-4a9b-a821-5c94a5cc8241-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a80c90f1-2746-4a9b-a821-5c94a5cc8241-routing-overlay-volume\") pod \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\" (UID: \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\") "
Oct 30 20:14:24.533196 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.533125    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a80c90f1-2746-4a9b-a821-5c94a5cc8241-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a80c90f1-2746-4a9b-a821-5c94a5cc8241-provider-credentials\") pod \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\" (UID: \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\") "
Oct 30 20:14:24.533335 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.533202    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a80c90f1-2746-4a9b-a821-5c94a5cc8241-kube-api-access-bnvsj\" (UniqueName: \"kubernetes.io/projected/a80c90f1-2746-4a9b-a821-5c94a5cc8241-kube-api-access-bnvsj\") pod \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\" (UID: \"a80c90f1-2746-4a9b-a821-5c94a5cc8241\") "
Oct 30 20:14:24.533414 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.533386    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80c90f1-2746-4a9b-a821-5c94a5cc8241-plugins-config-volume" pod "a80c90f1-2746-4a9b-a821-5c94a5cc8241" (UID: "a80c90f1-2746-4a9b-a821-5c94a5cc8241"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:24.533500 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.533472    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80c90f1-2746-4a9b-a821-5c94a5cc8241-routing-overlay-volume" pod "a80c90f1-2746-4a9b-a821-5c94a5cc8241" (UID: "a80c90f1-2746-4a9b-a821-5c94a5cc8241"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:24.533765 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.533745    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80c90f1-2746-4a9b-a821-5c94a5cc8241-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:24.533765 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.533764    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a80c90f1-2746-4a9b-a821-5c94a5cc8241-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:24.535529 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.535497    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80c90f1-2746-4a9b-a821-5c94a5cc8241-provider-credentials" pod "a80c90f1-2746-4a9b-a821-5c94a5cc8241" (UID: "a80c90f1-2746-4a9b-a821-5c94a5cc8241"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:24.535623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.535604    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80c90f1-2746-4a9b-a821-5c94a5cc8241-kube-api-access-bnvsj" pod "a80c90f1-2746-4a9b-a821-5c94a5cc8241" (UID: "a80c90f1-2746-4a9b-a821-5c94a5cc8241"). InnerVolumeSpecName "kube-api-access-bnvsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:24.635002 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.634969    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a80c90f1-2746-4a9b-a821-5c94a5cc8241-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:24.635002 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:24.634997    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnvsj\" (UniqueName: \"kubernetes.io/projected/a80c90f1-2746-4a9b-a821-5c94a5cc8241-kube-api-access-bnvsj\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:25.127731 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:25.127696    6030 generic.go:356] "Generic (PLEG): container finished" podID="a80c90f1-2746-4a9b-a821-5c94a5cc8241" containerID="98322070e921a20f398cd6e5aa4bf90cb1ae0a14861cc52b9846165a92a473e6" exitCode=0
Oct 30 20:14:25.128234 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:25.127754    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"
Oct 30 20:14:25.128234 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:25.127776    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h" event={"ID":"a80c90f1-2746-4a9b-a821-5c94a5cc8241","Type":"ContainerDied","Data":"98322070e921a20f398cd6e5aa4bf90cb1ae0a14861cc52b9846165a92a473e6"}
Oct 30 20:14:25.128234 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:25.127857    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h" event={"ID":"a80c90f1-2746-4a9b-a821-5c94a5cc8241","Type":"ContainerDied","Data":"90c84e32c7c526f1537f34e1787bb70ba626cc8ebbff4e555e0812a4ee9da282"}
Oct 30 20:14:25.128234 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:25.127880    6030 scope.go:122] "RemoveContainer" containerID="98322070e921a20f398cd6e5aa4bf90cb1ae0a14861cc52b9846165a92a473e6"
Oct 30 20:14:25.139165 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:25.139143    6030 scope.go:122] "RemoveContainer" containerID="98322070e921a20f398cd6e5aa4bf90cb1ae0a14861cc52b9846165a92a473e6"
Oct 30 20:14:25.139432 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:14:25.139411    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98322070e921a20f398cd6e5aa4bf90cb1ae0a14861cc52b9846165a92a473e6\": container with ID starting with 98322070e921a20f398cd6e5aa4bf90cb1ae0a14861cc52b9846165a92a473e6 not found: ID does not exist" containerID="98322070e921a20f398cd6e5aa4bf90cb1ae0a14861cc52b9846165a92a473e6"
Oct 30 20:14:25.139512 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:25.139447    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98322070e921a20f398cd6e5aa4bf90cb1ae0a14861cc52b9846165a92a473e6"} err="failed to get container status \"98322070e921a20f398cd6e5aa4bf90cb1ae0a14861cc52b9846165a92a473e6\": rpc error: code = NotFound desc = could not find container \"98322070e921a20f398cd6e5aa4bf90cb1ae0a14861cc52b9846165a92a473e6\": container with ID starting with 98322070e921a20f398cd6e5aa4bf90cb1ae0a14861cc52b9846165a92a473e6 not found: ID does not exist"
Oct 30 20:14:25.154598 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:25.154579    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"]
Oct 30 20:14:25.161604 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:25.161582    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fgfq6h"]
Oct 30 20:14:25.203000 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:25.202973    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49d09bca-f401-48bf-b2c2-8395bd8c2cf3" path="/var/lib/kubelet/pods/49d09bca-f401-48bf-b2c2-8395bd8c2cf3/volumes"
Oct 30 20:14:25.203518 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:25.203500    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a80c90f1-2746-4a9b-a821-5c94a5cc8241" path="/var/lib/kubelet/pods/a80c90f1-2746-4a9b-a821-5c94a5cc8241/volumes"
Oct 30 20:14:26.240145 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:26.240117    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"]
Oct 30 20:14:26.240657 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:26.240331    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn" podUID="af2af060-ec91-4d0f-b61d-cfec82fc4e2f" containerName="istio-proxy" containerID="cri-o://fbfc3759e8d67a9ce6e8f418ea9a5c48c4309dc028fd9df23595308dd17fc1a1" gracePeriod=30
Oct 30 20:14:31.514737 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.514676    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 20:14:31.631132 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.631103    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istiod-ca-cert\") pod \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") "
Oct 30 20:14:31.631346 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.631143    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-workload-socket\") pod \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") "
Oct 30 20:14:31.631346 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.631188    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-kube-api-access-j2vvk\" (UniqueName: \"kubernetes.io/projected/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-kube-api-access-j2vvk\") pod \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") "
Oct 30 20:14:31.631346 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.631232    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-envoy\") pod \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") "
Oct 30 20:14:31.631465 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.631348    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-workload-certs\") pod \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") "
Oct 30 20:14:31.631465 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.631393    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-data\") pod \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") "
Oct 30 20:14:31.631465 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.631451    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-podinfo\") pod \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") "
Oct 30 20:14:31.631610 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.631479    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-credential-socket\") pod \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") "
Oct 30 20:14:31.631610 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.631522    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-token\" (UniqueName: \"kubernetes.io/projected/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-token\") pod \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\" (UID: \"af2af060-ec91-4d0f-b61d-cfec82fc4e2f\") "
Oct 30 20:14:31.631720 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.631473    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-workload-socket" pod "af2af060-ec91-4d0f-b61d-cfec82fc4e2f" (UID: "af2af060-ec91-4d0f-b61d-cfec82fc4e2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:31.631720 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.631542    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istiod-ca-cert" pod "af2af060-ec91-4d0f-b61d-cfec82fc4e2f" (UID: "af2af060-ec91-4d0f-b61d-cfec82fc4e2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:31.631720 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.631653    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-workload-certs" pod "af2af060-ec91-4d0f-b61d-cfec82fc4e2f" (UID: "af2af060-ec91-4d0f-b61d-cfec82fc4e2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:31.631720 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.631668    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-data" pod "af2af060-ec91-4d0f-b61d-cfec82fc4e2f" (UID: "af2af060-ec91-4d0f-b61d-cfec82fc4e2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:31.632002 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.631837    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-credential-socket" pod "af2af060-ec91-4d0f-b61d-cfec82fc4e2f" (UID: "af2af060-ec91-4d0f-b61d-cfec82fc4e2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:31.632237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.632181    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:31.632237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.632206    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:31.632237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.632221    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:31.632237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.632233    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:31.632237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.632242    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:31.633768 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.633744    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-kube-api-access-j2vvk" pod "af2af060-ec91-4d0f-b61d-cfec82fc4e2f" (UID: "af2af060-ec91-4d0f-b61d-cfec82fc4e2f"). InnerVolumeSpecName "kube-api-access-j2vvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:31.633859 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.633772    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-podinfo" pod "af2af060-ec91-4d0f-b61d-cfec82fc4e2f" (UID: "af2af060-ec91-4d0f-b61d-cfec82fc4e2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:14:31.634170 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.634145    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-envoy" pod "af2af060-ec91-4d0f-b61d-cfec82fc4e2f" (UID: "af2af060-ec91-4d0f-b61d-cfec82fc4e2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:31.634247 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.634203    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-token" pod "af2af060-ec91-4d0f-b61d-cfec82fc4e2f" (UID: "af2af060-ec91-4d0f-b61d-cfec82fc4e2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:31.733803 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.733762    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:31.733803 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.733804    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:31.734017 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.733819    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:31.734017 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.733833    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2vvk\" (UniqueName: \"kubernetes.io/projected/af2af060-ec91-4d0f-b61d-cfec82fc4e2f-kube-api-access-j2vvk\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:31.755576 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.755545    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"]
Oct 30 20:14:31.783287 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:31.783206    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"]
Oct 30 20:14:32.183916 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:32.183877    6030 generic.go:356] "Generic (PLEG): container finished" podID="af2af060-ec91-4d0f-b61d-cfec82fc4e2f" containerID="fbfc3759e8d67a9ce6e8f418ea9a5c48c4309dc028fd9df23595308dd17fc1a1" exitCode=0
Oct 30 20:14:32.184139 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:32.183943    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"
Oct 30 20:14:32.184237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:32.184132    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn" event={"ID":"af2af060-ec91-4d0f-b61d-cfec82fc4e2f","Type":"ContainerDied","Data":"fbfc3759e8d67a9ce6e8f418ea9a5c48c4309dc028fd9df23595308dd17fc1a1"}
Oct 30 20:14:32.184237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:32.184203    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn" event={"ID":"af2af060-ec91-4d0f-b61d-cfec82fc4e2f","Type":"ContainerDied","Data":"68505c52f8038037234ab36dd723a8ff216601fea27866b63c748e3b41c1e01a"}
Oct 30 20:14:32.184340 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:32.184241    6030 scope.go:122] "RemoveContainer" containerID="fbfc3759e8d67a9ce6e8f418ea9a5c48c4309dc028fd9df23595308dd17fc1a1"
Oct 30 20:14:32.196737 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:32.196718    6030 scope.go:122] "RemoveContainer" containerID="fbfc3759e8d67a9ce6e8f418ea9a5c48c4309dc028fd9df23595308dd17fc1a1"
Oct 30 20:14:32.196998 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:14:32.196979    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbfc3759e8d67a9ce6e8f418ea9a5c48c4309dc028fd9df23595308dd17fc1a1\": container with ID starting with fbfc3759e8d67a9ce6e8f418ea9a5c48c4309dc028fd9df23595308dd17fc1a1 not found: ID does not exist" containerID="fbfc3759e8d67a9ce6e8f418ea9a5c48c4309dc028fd9df23595308dd17fc1a1"
Oct 30 20:14:32.197109 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:32.197007    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbfc3759e8d67a9ce6e8f418ea9a5c48c4309dc028fd9df23595308dd17fc1a1"} err="failed to get container status \"fbfc3759e8d67a9ce6e8f418ea9a5c48c4309dc028fd9df23595308dd17fc1a1\": rpc error: code = NotFound desc = could not find container \"fbfc3759e8d67a9ce6e8f418ea9a5c48c4309dc028fd9df23595308dd17fc1a1\": container with ID starting with fbfc3759e8d67a9ce6e8f418ea9a5c48c4309dc028fd9df23595308dd17fc1a1 not found: ID does not exist"
Oct 30 20:14:32.213232 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:32.213197    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"]
Oct 30 20:14:32.219151 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:32.219122    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn"]
Oct 30 20:14:32.464223 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:32.464138    6030 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:14:32.464223 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:32.464213    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-g8rtn" podUID="af2af060-ec91-4d0f-b61d-cfec82fc4e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:14:33.202641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:33.202613    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af2af060-ec91-4d0f-b61d-cfec82fc4e2f" path="/var/lib/kubelet/pods/af2af060-ec91-4d0f-b61d-cfec82fc4e2f/volumes"
Oct 30 20:14:34.290599 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:14:34.290547    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:14:36.755936 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:36.755877    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz" podUID="24e84582-fba1-41f5-b4e1-bd19d13f33db" containerName="payload-processing" containerID="cri-o://8cedd609f1e98381906cbcb65d7b8171aea0784d33ed6265c7997eda1f156629" gracePeriod=25
Oct 30 20:14:36.783925 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:36.783858    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl" podUID="d426ce34-85fa-4a6f-8d0e-dfe7ead02afb" containerName="payload-processing" containerID="cri-o://33efe897d29d2f4a2e5d9bd7773a1bb43fcc62663850654f8a123e6c4719bc38" gracePeriod=25
Oct 30 20:14:37.084935 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.084910    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:37.115925 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.115900    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:37.198899 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.198867    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e84582-fba1-41f5-b4e1-bd19d13f33db-kube-api-access-f2dd6\" (UniqueName: \"kubernetes.io/projected/24e84582-fba1-41f5-b4e1-bd19d13f33db-kube-api-access-f2dd6\") pod \"24e84582-fba1-41f5-b4e1-bd19d13f33db\" (UID: \"24e84582-fba1-41f5-b4e1-bd19d13f33db\") "
Oct 30 20:14:37.199108 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.198952    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-routing-overlay-volume\") pod \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\" (UID: \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\") "
Oct 30 20:14:37.199108 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.199083    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-plugins-config-volume\") pod \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\" (UID: \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\") "
Oct 30 20:14:37.199241 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.199158    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-kube-api-access-k5km4\" (UniqueName: \"kubernetes.io/projected/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-kube-api-access-k5km4\") pod \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\" (UID: \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\") "
Oct 30 20:14:37.199241 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.199198    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24e84582-fba1-41f5-b4e1-bd19d13f33db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e84582-fba1-41f5-b4e1-bd19d13f33db-plugins-config-volume\") pod \"24e84582-fba1-41f5-b4e1-bd19d13f33db\" (UID: \"24e84582-fba1-41f5-b4e1-bd19d13f33db\") "
Oct 30 20:14:37.199349 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.199280    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24e84582-fba1-41f5-b4e1-bd19d13f33db-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24e84582-fba1-41f5-b4e1-bd19d13f33db-routing-overlay-volume\") pod \"24e84582-fba1-41f5-b4e1-bd19d13f33db\" (UID: \"24e84582-fba1-41f5-b4e1-bd19d13f33db\") "
Oct 30 20:14:37.199349 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.199338    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-provider-credentials\") pod \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\" (UID: \"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb\") "
Oct 30 20:14:37.199464 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.199374    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e84582-fba1-41f5-b4e1-bd19d13f33db-provider-credentials\" (UniqueName: \"kubernetes.io/projected/24e84582-fba1-41f5-b4e1-bd19d13f33db-provider-credentials\") pod \"24e84582-fba1-41f5-b4e1-bd19d13f33db\" (UID: \"24e84582-fba1-41f5-b4e1-bd19d13f33db\") "
Oct 30 20:14:37.199464 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.199431    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-routing-overlay-volume" pod "d426ce34-85fa-4a6f-8d0e-dfe7ead02afb" (UID: "d426ce34-85fa-4a6f-8d0e-dfe7ead02afb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:37.199741 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.199717    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e84582-fba1-41f5-b4e1-bd19d13f33db-plugins-config-volume" pod "24e84582-fba1-41f5-b4e1-bd19d13f33db" (UID: "24e84582-fba1-41f5-b4e1-bd19d13f33db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:37.199741 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.199726    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-plugins-config-volume" pod "d426ce34-85fa-4a6f-8d0e-dfe7ead02afb" (UID: "d426ce34-85fa-4a6f-8d0e-dfe7ead02afb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:37.199879 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.199749    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e84582-fba1-41f5-b4e1-bd19d13f33db-routing-overlay-volume" pod "24e84582-fba1-41f5-b4e1-bd19d13f33db" (UID: "24e84582-fba1-41f5-b4e1-bd19d13f33db"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:37.199879 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.199865    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:37.199991 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.199883    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e84582-fba1-41f5-b4e1-bd19d13f33db-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:37.199991 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.199897    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24e84582-fba1-41f5-b4e1-bd19d13f33db-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:37.199991 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.199910    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:37.201931 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.201904    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-kube-api-access-k5km4" pod "d426ce34-85fa-4a6f-8d0e-dfe7ead02afb" (UID: "d426ce34-85fa-4a6f-8d0e-dfe7ead02afb"). InnerVolumeSpecName "kube-api-access-k5km4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:37.202119 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.202093    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-provider-credentials" pod "d426ce34-85fa-4a6f-8d0e-dfe7ead02afb" (UID: "d426ce34-85fa-4a6f-8d0e-dfe7ead02afb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:37.202258 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.202234    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e84582-fba1-41f5-b4e1-bd19d13f33db-kube-api-access-f2dd6" pod "24e84582-fba1-41f5-b4e1-bd19d13f33db" (UID: "24e84582-fba1-41f5-b4e1-bd19d13f33db"). InnerVolumeSpecName "kube-api-access-f2dd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:37.202459 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.202387    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e84582-fba1-41f5-b4e1-bd19d13f33db-provider-credentials" pod "24e84582-fba1-41f5-b4e1-bd19d13f33db" (UID: "24e84582-fba1-41f5-b4e1-bd19d13f33db"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:37.222231 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.222211    6030 generic.go:356] "Generic (PLEG): container finished" podID="24e84582-fba1-41f5-b4e1-bd19d13f33db" containerID="8cedd609f1e98381906cbcb65d7b8171aea0784d33ed6265c7997eda1f156629" exitCode=0
Oct 30 20:14:37.222326 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.222311    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"
Oct 30 20:14:37.222372 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.222342    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz" event={"ID":"24e84582-fba1-41f5-b4e1-bd19d13f33db","Type":"ContainerDied","Data":"8cedd609f1e98381906cbcb65d7b8171aea0784d33ed6265c7997eda1f156629"}
Oct 30 20:14:37.222413 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.222397    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz" event={"ID":"24e84582-fba1-41f5-b4e1-bd19d13f33db","Type":"ContainerDied","Data":"16c61d01414f25c921ca73bf4be720edd36234ef8f4fd1bedfbd966f430c49cb"}
Oct 30 20:14:37.222459 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.222430    6030 scope.go:122] "RemoveContainer" containerID="8cedd609f1e98381906cbcb65d7b8171aea0784d33ed6265c7997eda1f156629"
Oct 30 20:14:37.224245 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.224223    6030 generic.go:356] "Generic (PLEG): container finished" podID="d426ce34-85fa-4a6f-8d0e-dfe7ead02afb" containerID="33efe897d29d2f4a2e5d9bd7773a1bb43fcc62663850654f8a123e6c4719bc38" exitCode=0
Oct 30 20:14:37.224349 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.224249    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl" event={"ID":"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb","Type":"ContainerDied","Data":"33efe897d29d2f4a2e5d9bd7773a1bb43fcc62663850654f8a123e6c4719bc38"}
Oct 30 20:14:37.224349 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.224296    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl" event={"ID":"d426ce34-85fa-4a6f-8d0e-dfe7ead02afb","Type":"ContainerDied","Data":"6079fbcececdfa9e6ffe89fcc45a8c868abe2a2530ef79cfbc940a9f8a15fcc4"}
Oct 30 20:14:37.224349 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.224313    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"
Oct 30 20:14:37.232777 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.232752    6030 scope.go:122] "RemoveContainer" containerID="8cedd609f1e98381906cbcb65d7b8171aea0784d33ed6265c7997eda1f156629"
Oct 30 20:14:37.233483 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:14:37.233058    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cedd609f1e98381906cbcb65d7b8171aea0784d33ed6265c7997eda1f156629\": container with ID starting with 8cedd609f1e98381906cbcb65d7b8171aea0784d33ed6265c7997eda1f156629 not found: ID does not exist" containerID="8cedd609f1e98381906cbcb65d7b8171aea0784d33ed6265c7997eda1f156629"
Oct 30 20:14:37.233483 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.233093    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cedd609f1e98381906cbcb65d7b8171aea0784d33ed6265c7997eda1f156629"} err="failed to get container status \"8cedd609f1e98381906cbcb65d7b8171aea0784d33ed6265c7997eda1f156629\": rpc error: code = NotFound desc = could not find container \"8cedd609f1e98381906cbcb65d7b8171aea0784d33ed6265c7997eda1f156629\": container with ID starting with 8cedd609f1e98381906cbcb65d7b8171aea0784d33ed6265c7997eda1f156629 not found: ID does not exist"
Oct 30 20:14:37.233483 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.233115    6030 scope.go:122] "RemoveContainer" containerID="33efe897d29d2f4a2e5d9bd7773a1bb43fcc62663850654f8a123e6c4719bc38"
Oct 30 20:14:37.245207 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.245177    6030 scope.go:122] "RemoveContainer" containerID="33efe897d29d2f4a2e5d9bd7773a1bb43fcc62663850654f8a123e6c4719bc38"
Oct 30 20:14:37.245528 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:14:37.245505    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33efe897d29d2f4a2e5d9bd7773a1bb43fcc62663850654f8a123e6c4719bc38\": container with ID starting with 33efe897d29d2f4a2e5d9bd7773a1bb43fcc62663850654f8a123e6c4719bc38 not found: ID does not exist" containerID="33efe897d29d2f4a2e5d9bd7773a1bb43fcc62663850654f8a123e6c4719bc38"
Oct 30 20:14:37.245627 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.245534    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33efe897d29d2f4a2e5d9bd7773a1bb43fcc62663850654f8a123e6c4719bc38"} err="failed to get container status \"33efe897d29d2f4a2e5d9bd7773a1bb43fcc62663850654f8a123e6c4719bc38\": rpc error: code = NotFound desc = could not find container \"33efe897d29d2f4a2e5d9bd7773a1bb43fcc62663850654f8a123e6c4719bc38\": container with ID starting with 33efe897d29d2f4a2e5d9bd7773a1bb43fcc62663850654f8a123e6c4719bc38 not found: ID does not exist"
Oct 30 20:14:37.249528 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.249504    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"]
Oct 30 20:14:37.261109 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.261087    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8k7wnz"]
Oct 30 20:14:37.276750 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.276724    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"]
Oct 30 20:14:37.280435 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.280413    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6nw7pl"]
Oct 30 20:14:37.301697 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.301629    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:37.301697 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.301650    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24e84582-fba1-41f5-b4e1-bd19d13f33db-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:37.301697 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.301660    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2dd6\" (UniqueName: \"kubernetes.io/projected/24e84582-fba1-41f5-b4e1-bd19d13f33db-kube-api-access-f2dd6\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:37.301697 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:37.301673    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5km4\" (UniqueName: \"kubernetes.io/projected/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb-kube-api-access-k5km4\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:39.204075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:39.204023    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24e84582-fba1-41f5-b4e1-bd19d13f33db" path="/var/lib/kubelet/pods/24e84582-fba1-41f5-b4e1-bd19d13f33db/volumes"
Oct 30 20:14:39.204472 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:39.204457    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d426ce34-85fa-4a6f-8d0e-dfe7ead02afb" path="/var/lib/kubelet/pods/d426ce34-85fa-4a6f-8d0e-dfe7ead02afb/volumes"
Oct 30 20:14:45.332303 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.332268    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"]
Oct 30 20:14:45.334778 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.334742    6030 cpu_manager.go:414] "containerMap: removing container" podUID="a80c90f1-2746-4a9b-a821-5c94a5cc8241" containerName="payload-processing"
Oct 30 20:14:45.334959 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.334942    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80c90f1-2746-4a9b-a821-5c94a5cc8241" containerName="payload-processing"
Oct 30 20:14:45.335110 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.335094    6030 cpu_manager.go:414] "containerMap: removing container" podUID="d426ce34-85fa-4a6f-8d0e-dfe7ead02afb" containerName="payload-processing"
Oct 30 20:14:45.335217 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.335204    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d426ce34-85fa-4a6f-8d0e-dfe7ead02afb" containerName="payload-processing"
Oct 30 20:14:45.335347 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.335335    6030 cpu_manager.go:414] "containerMap: removing container" podUID="039cd584-b9e1-4b28-b0d5-5c482cbdaad3" containerName="istio-proxy"
Oct 30 20:14:45.335444 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.335433    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="039cd584-b9e1-4b28-b0d5-5c482cbdaad3" containerName="istio-proxy"
Oct 30 20:14:45.335540 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.335529    6030 cpu_manager.go:414] "containerMap: removing container" podUID="49d09bca-f401-48bf-b2c2-8395bd8c2cf3" containerName="payload-processing"
Oct 30 20:14:45.336071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.336036    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49d09bca-f401-48bf-b2c2-8395bd8c2cf3" containerName="payload-processing"
Oct 30 20:14:45.336261 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.336246    6030 cpu_manager.go:414] "containerMap: removing container" podUID="24e84582-fba1-41f5-b4e1-bd19d13f33db" containerName="payload-processing"
Oct 30 20:14:45.336361 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.336348    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e84582-fba1-41f5-b4e1-bd19d13f33db" containerName="payload-processing"
Oct 30 20:14:45.336485 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.336473    6030 cpu_manager.go:414] "containerMap: removing container" podUID="af2af060-ec91-4d0f-b61d-cfec82fc4e2f" containerName="istio-proxy"
Oct 30 20:14:45.336574 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.336563    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af2af060-ec91-4d0f-b61d-cfec82fc4e2f" containerName="istio-proxy"
Oct 30 20:14:45.336824 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.336808    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80c90f1-2746-4a9b-a821-5c94a5cc8241" containerName="payload-processing"
Oct 30 20:14:45.336939 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.336926    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="d426ce34-85fa-4a6f-8d0e-dfe7ead02afb" containerName="payload-processing"
Oct 30 20:14:45.337128 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.337113    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="039cd584-b9e1-4b28-b0d5-5c482cbdaad3" containerName="istio-proxy"
Oct 30 20:14:45.337300 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.337276    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="49d09bca-f401-48bf-b2c2-8395bd8c2cf3" containerName="payload-processing"
Oct 30 20:14:45.337404 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.337394    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e84582-fba1-41f5-b4e1-bd19d13f33db" containerName="payload-processing"
Oct 30 20:14:45.337545 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.337534    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="af2af060-ec91-4d0f-b61d-cfec82fc4e2f" containerName="istio-proxy"
Oct 30 20:14:45.358108 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.351732    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"]
Oct 30 20:14:45.358108 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.351854    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:14:45.377096 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.376377    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c06e1d87a941\""
Oct 30 20:14:45.377096 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.376587    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9d45aadaaa09\""
Oct 30 20:14:45.377096 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.376624    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 20:14:45.377096 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.376668    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 20:14:45.377096 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.376710    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 20:14:45.377972 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.377955    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 20:14:45.378067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.378001    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-mqtgc\""
Oct 30 20:14:45.429139 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.429109    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c79d7c3c-99cb-44c0-88c6-23daef726eec-routing-overlay-volume\") pod \"payload-processing-external-model-58977ddb8f-w62d5\" (UID: \"c79d7c3c-99cb-44c0-88c6-23daef726eec\") " pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:14:45.429318 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.429299    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c79d7c3c-99cb-44c0-88c6-23daef726eec-provider-credentials\") pod \"payload-processing-external-model-58977ddb8f-w62d5\" (UID: \"c79d7c3c-99cb-44c0-88c6-23daef726eec\") " pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:14:45.429363 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.429343    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c79d7c3c-99cb-44c0-88c6-23daef726eec-plugins-config-volume\") pod \"payload-processing-external-model-58977ddb8f-w62d5\" (UID: \"c79d7c3c-99cb-44c0-88c6-23daef726eec\") " pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:14:45.429466 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.429442    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9vpj\" (UniqueName: \"kubernetes.io/projected/c79d7c3c-99cb-44c0-88c6-23daef726eec-kube-api-access-k9vpj\") pod \"payload-processing-external-model-58977ddb8f-w62d5\" (UID: \"c79d7c3c-99cb-44c0-88c6-23daef726eec\") " pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:14:45.531212 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.531175    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c79d7c3c-99cb-44c0-88c6-23daef726eec-provider-credentials\") pod \"payload-processing-external-model-58977ddb8f-w62d5\" (UID: \"c79d7c3c-99cb-44c0-88c6-23daef726eec\") " pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:14:45.531379 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.531227    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c79d7c3c-99cb-44c0-88c6-23daef726eec-plugins-config-volume\") pod \"payload-processing-external-model-58977ddb8f-w62d5\" (UID: \"c79d7c3c-99cb-44c0-88c6-23daef726eec\") " pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:14:45.531379 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.531269    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9vpj\" (UniqueName: \"kubernetes.io/projected/c79d7c3c-99cb-44c0-88c6-23daef726eec-kube-api-access-k9vpj\") pod \"payload-processing-external-model-58977ddb8f-w62d5\" (UID: \"c79d7c3c-99cb-44c0-88c6-23daef726eec\") " pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:14:45.531379 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.531329    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c79d7c3c-99cb-44c0-88c6-23daef726eec-routing-overlay-volume\") pod \"payload-processing-external-model-58977ddb8f-w62d5\" (UID: \"c79d7c3c-99cb-44c0-88c6-23daef726eec\") " pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:14:45.537348 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.532489    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c79d7c3c-99cb-44c0-88c6-23daef726eec-routing-overlay-volume\") pod \"payload-processing-external-model-58977ddb8f-w62d5\" (UID: \"c79d7c3c-99cb-44c0-88c6-23daef726eec\") " pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:14:45.537348 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.532668    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c79d7c3c-99cb-44c0-88c6-23daef726eec-plugins-config-volume\") pod \"payload-processing-external-model-58977ddb8f-w62d5\" (UID: \"c79d7c3c-99cb-44c0-88c6-23daef726eec\") " pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:14:45.545679 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.545641    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c79d7c3c-99cb-44c0-88c6-23daef726eec-provider-credentials\") pod \"payload-processing-external-model-58977ddb8f-w62d5\" (UID: \"c79d7c3c-99cb-44c0-88c6-23daef726eec\") " pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:14:45.549958 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.549934    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9vpj\" (UniqueName: \"kubernetes.io/projected/c79d7c3c-99cb-44c0-88c6-23daef726eec-kube-api-access-k9vpj\") pod \"payload-processing-external-model-58977ddb8f-w62d5\" (UID: \"c79d7c3c-99cb-44c0-88c6-23daef726eec\") " pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:14:45.676812 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.676777    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:14:45.998337 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:45.998306    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"]
Oct 30 20:14:46.000104 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:14:46.000071    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79d7c3c_99cb_44c0_88c6_23daef726eec.slice/crio-19575147378a918a337ad0ffbaf15f7b7da003bf62fcd892630aaa4825441a48 WatchSource:0}: Error finding container 19575147378a918a337ad0ffbaf15f7b7da003bf62fcd892630aaa4825441a48: Status 404 returned error can't find the container with id 19575147378a918a337ad0ffbaf15f7b7da003bf62fcd892630aaa4825441a48
Oct 30 20:14:46.298247 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:46.298156    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5" event={"ID":"c79d7c3c-99cb-44c0-88c6-23daef726eec","Type":"ContainerStarted","Data":"5c0aaa4837ee862fd43bb18392cae045d57d6eaa9c722aa7bd1de5891bad051c"}
Oct 30 20:14:46.298247 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:46.298223    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5" event={"ID":"c79d7c3c-99cb-44c0-88c6-23daef726eec","Type":"ContainerStarted","Data":"19575147378a918a337ad0ffbaf15f7b7da003bf62fcd892630aaa4825441a48"}
Oct 30 20:14:46.298473 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:46.298366    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:14:46.321442 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:46.321390    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5" podStartSLOduration=1.3213770120000001 podStartE2EDuration="1.321377012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:46.320283752 +0000 UTC m=+2565.235890609" watchObservedRunningTime="2026-10-30 20:14:46.321377012 +0000 UTC m=+2565.236983867"
Oct 30 20:14:53.484292 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.484254    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"]
Oct 30 20:14:53.489501 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.489477    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"
Oct 30 20:14:53.505836 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.505813    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f63d20\""
Oct 30 20:14:53.506601 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.506577    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"]
Oct 30 20:14:53.513256 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.513221    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f63d20-dockercfg-2tppp\""
Oct 30 20:14:53.556678 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.556649    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlvpk\" (UniqueName: \"kubernetes.io/projected/a083cbc5-d30b-4103-a5d8-a695e84418f2-kube-api-access-dlvpk\") pod \"payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx\" (UID: \"a083cbc5-d30b-4103-a5d8-a695e84418f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"
Oct 30 20:14:53.556847 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.556801    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a083cbc5-d30b-4103-a5d8-a695e84418f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx\" (UID: \"a083cbc5-d30b-4103-a5d8-a695e84418f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"
Oct 30 20:14:53.564726 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.564696    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"]
Oct 30 20:14:53.569457 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.569439    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"
Oct 30 20:14:53.585632 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.585605    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"]
Oct 30 20:14:53.697070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.674105    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6md5q\" (UniqueName: \"kubernetes.io/projected/914369a4-b049-4fa2-ab17-f31d0b21b083-kube-api-access-6md5q\") pod \"payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn\" (UID: \"914369a4-b049-4fa2-ab17-f31d0b21b083\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"
Oct 30 20:14:53.728012 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.727975    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvpk\" (UniqueName: \"kubernetes.io/projected/a083cbc5-d30b-4103-a5d8-a695e84418f2-kube-api-access-dlvpk\") pod \"payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx\" (UID: \"a083cbc5-d30b-4103-a5d8-a695e84418f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"
Oct 30 20:14:53.728575 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.728274    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914369a4-b049-4fa2-ab17-f31d0b21b083-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn\" (UID: \"914369a4-b049-4fa2-ab17-f31d0b21b083\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"
Oct 30 20:14:53.728948 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.728898    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a083cbc5-d30b-4103-a5d8-a695e84418f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx\" (UID: \"a083cbc5-d30b-4103-a5d8-a695e84418f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"
Oct 30 20:14:53.730512 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.730430    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a083cbc5-d30b-4103-a5d8-a695e84418f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx\" (UID: \"a083cbc5-d30b-4103-a5d8-a695e84418f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"
Oct 30 20:14:53.758155 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.758081    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvpk\" (UniqueName: \"kubernetes.io/projected/a083cbc5-d30b-4103-a5d8-a695e84418f2-kube-api-access-dlvpk\") pod \"payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx\" (UID: \"a083cbc5-d30b-4103-a5d8-a695e84418f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"
Oct 30 20:14:53.804487 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.804458    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"
Oct 30 20:14:53.830593 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.830558    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6md5q\" (UniqueName: \"kubernetes.io/projected/914369a4-b049-4fa2-ab17-f31d0b21b083-kube-api-access-6md5q\") pod \"payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn\" (UID: \"914369a4-b049-4fa2-ab17-f31d0b21b083\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"
Oct 30 20:14:53.830763 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.830735    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914369a4-b049-4fa2-ab17-f31d0b21b083-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn\" (UID: \"914369a4-b049-4fa2-ab17-f31d0b21b083\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"
Oct 30 20:14:53.831644 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.831620    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914369a4-b049-4fa2-ab17-f31d0b21b083-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn\" (UID: \"914369a4-b049-4fa2-ab17-f31d0b21b083\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"
Oct 30 20:14:53.845390 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.845362    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6md5q\" (UniqueName: \"kubernetes.io/projected/914369a4-b049-4fa2-ab17-f31d0b21b083-kube-api-access-6md5q\") pod \"payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn\" (UID: \"914369a4-b049-4fa2-ab17-f31d0b21b083\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"
Oct 30 20:14:53.889086 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:53.884680    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"
Oct 30 20:14:54.100135 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:54.099991    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"]
Oct 30 20:14:54.100499 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:14:54.100478    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda083cbc5_d30b_4103_a5d8_a695e84418f2.slice/crio-d6eda001a19ca1b08ad422f3b0f72cfad7afe04ed5acfb63fefc4414660e7029 WatchSource:0}: Error finding container d6eda001a19ca1b08ad422f3b0f72cfad7afe04ed5acfb63fefc4414660e7029: Status 404 returned error can't find the container with id d6eda001a19ca1b08ad422f3b0f72cfad7afe04ed5acfb63fefc4414660e7029
Oct 30 20:14:54.189417 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:14:54.189384    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914369a4_b049_4fa2_ab17_f31d0b21b083.slice/crio-fc45b9fc963d4069ade75c6a0912632832dbc470093f3f5468714721b7322676 WatchSource:0}: Error finding container fc45b9fc963d4069ade75c6a0912632832dbc470093f3f5468714721b7322676: Status 404 returned error can't find the container with id fc45b9fc963d4069ade75c6a0912632832dbc470093f3f5468714721b7322676
Oct 30 20:14:54.189540 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:54.189413    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"]
Oct 30 20:14:54.371172 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:54.371081    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx" event={"ID":"a083cbc5-d30b-4103-a5d8-a695e84418f2","Type":"ContainerStarted","Data":"201cbd61605c732ec6c398dbabbef339577c4dc904fd4fad394f858600939597"}
Oct 30 20:14:54.371172 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:54.371141    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx" event={"ID":"a083cbc5-d30b-4103-a5d8-a695e84418f2","Type":"ContainerStarted","Data":"d6eda001a19ca1b08ad422f3b0f72cfad7afe04ed5acfb63fefc4414660e7029"}
Oct 30 20:14:54.371409 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:54.371365    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"
Oct 30 20:14:54.372959 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:54.372929    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn" event={"ID":"914369a4-b049-4fa2-ab17-f31d0b21b083","Type":"ContainerStarted","Data":"d85bc515b363adbaa0cbd2436c54dbf44db95ca437d10091417283c49d8ace3a"}
Oct 30 20:14:54.373143 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:54.372979    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn" event={"ID":"914369a4-b049-4fa2-ab17-f31d0b21b083","Type":"ContainerStarted","Data":"fc45b9fc963d4069ade75c6a0912632832dbc470093f3f5468714721b7322676"}
Oct 30 20:14:54.373143 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:54.373006    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"
Oct 30 20:14:54.393723 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:54.393673    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx" podStartSLOduration=1.3936619160000001 podStartE2EDuration="1.393661916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:54.390566532 +0000 UTC m=+2573.306173388" watchObservedRunningTime="2026-10-30 20:14:54.393661916 +0000 UTC m=+2573.309269038"
Oct 30 20:14:54.414154 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:54.414100    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn" podStartSLOduration=1.414083718 podStartE2EDuration="1.414083718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:54.411183363 +0000 UTC m=+2573.326790215" watchObservedRunningTime="2026-10-30 20:14:54.414083718 +0000 UTC m=+2573.329690578"
Oct 30 20:14:57.309472 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:14:57.309442    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:15:00.282003 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.281965    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28"]
Oct 30 20:15:00.286849 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.286822    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28"
Oct 30 20:15:00.318458 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.318338    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 20:15:00.320287 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.320225    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca670d4-cee3-4066-80e6-8b3a5ca7783b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28\" (UID: \"4ca670d4-cee3-4066-80e6-8b3a5ca7783b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28"
Oct 30 20:15:00.320527 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.320504    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzlv5\" (UniqueName: \"kubernetes.io/projected/4ca670d4-cee3-4066-80e6-8b3a5ca7783b-kube-api-access-nzlv5\") pod \"maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28\" (UID: \"4ca670d4-cee3-4066-80e6-8b3a5ca7783b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28"
Oct 30 20:15:00.323410 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.323389    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2"]
Oct 30 20:15:00.329950 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.329930    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5"]
Oct 30 20:15:00.330148 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.330108    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2"
Oct 30 20:15:00.335799 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.335773    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5"
Oct 30 20:15:00.336508 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.336484    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28"]
Oct 30 20:15:00.359014 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.358941    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2"]
Oct 30 20:15:00.363442 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.363409    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5"]
Oct 30 20:15:00.388015 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.387987    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8"]
Oct 30 20:15:00.392398 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.392345    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8"
Oct 30 20:15:00.393107 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.393083    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv"]
Oct 30 20:15:00.398945 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.398919    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv"
Oct 30 20:15:00.417894 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.417871    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv"]
Oct 30 20:15:00.423105 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.423076    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca670d4-cee3-4066-80e6-8b3a5ca7783b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28\" (UID: \"4ca670d4-cee3-4066-80e6-8b3a5ca7783b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28"
Oct 30 20:15:00.423258 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.423138    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5\" (UID: \"9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5"
Oct 30 20:15:00.423258 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.423177    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xc6k\" (UniqueName: \"kubernetes.io/projected/9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b-kube-api-access-6xc6k\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5\" (UID: \"9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5"
Oct 30 20:15:00.423389 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.423285    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/461521b6-3031-46d9-8e54-765c46db9983-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2\" (UID: \"461521b6-3031-46d9-8e54-765c46db9983\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2"
Oct 30 20:15:00.423389 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.423335    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzlv5\" (UniqueName: \"kubernetes.io/projected/4ca670d4-cee3-4066-80e6-8b3a5ca7783b-kube-api-access-nzlv5\") pod \"maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28\" (UID: \"4ca670d4-cee3-4066-80e6-8b3a5ca7783b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28"
Oct 30 20:15:00.423389 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.423384    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4szcq\" (UniqueName: \"kubernetes.io/projected/461521b6-3031-46d9-8e54-765c46db9983-kube-api-access-4szcq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2\" (UID: \"461521b6-3031-46d9-8e54-765c46db9983\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2"
Oct 30 20:15:00.424757 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.424729    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca670d4-cee3-4066-80e6-8b3a5ca7783b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28\" (UID: \"4ca670d4-cee3-4066-80e6-8b3a5ca7783b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28"
Oct 30 20:15:00.431102 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.431019    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8"]
Oct 30 20:15:00.443715 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.443686    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzlv5\" (UniqueName: \"kubernetes.io/projected/4ca670d4-cee3-4066-80e6-8b3a5ca7783b-kube-api-access-nzlv5\") pod \"maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28\" (UID: \"4ca670d4-cee3-4066-80e6-8b3a5ca7783b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28"
Oct 30 20:15:00.457799 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.457770    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t"]
Oct 30 20:15:00.468120 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.468092    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t"
Oct 30 20:15:00.468916 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.468888    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7"]
Oct 30 20:15:00.474559 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.474539    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t"]
Oct 30 20:15:00.474689 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.474635    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7"
Oct 30 20:15:00.487804 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.487749    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7"]
Oct 30 20:15:00.498463 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.498437    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z"]
Oct 30 20:15:00.504011 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.503988    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z"
Oct 30 20:15:00.525575 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.525541    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4szcq\" (UniqueName: \"kubernetes.io/projected/461521b6-3031-46d9-8e54-765c46db9983-kube-api-access-4szcq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2\" (UID: \"461521b6-3031-46d9-8e54-765c46db9983\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2"
Oct 30 20:15:00.525785 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.525766    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5\" (UID: \"9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5"
Oct 30 20:15:00.525851 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.525803    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4z97\" (UniqueName: \"kubernetes.io/projected/75e2878b-a4f8-4180-a5c2-cb169475f2d7-kube-api-access-b4z97\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7\" (UID: \"75e2878b-a4f8-4180-a5c2-cb169475f2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7"
Oct 30 20:15:00.525851 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.525837    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xc6k\" (UniqueName: \"kubernetes.io/projected/9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b-kube-api-access-6xc6k\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5\" (UID: \"9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5"
Oct 30 20:15:00.525959 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.525883    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380956ae-205c-4814-aa2d-68aae67dd2cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-7pmtv\" (UID: \"380956ae-205c-4814-aa2d-68aae67dd2cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv"
Oct 30 20:15:00.525959 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.525920    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59l2f\" (UniqueName: \"kubernetes.io/projected/380956ae-205c-4814-aa2d-68aae67dd2cc-kube-api-access-59l2f\") pod \"maas-api-key-cleanup-29889855-7pmtv\" (UID: \"380956ae-205c-4814-aa2d-68aae67dd2cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv"
Oct 30 20:15:00.526109 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.525960    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e2878b-a4f8-4180-a5c2-cb169475f2d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7\" (UID: \"75e2878b-a4f8-4180-a5c2-cb169475f2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7"
Oct 30 20:15:00.526109 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.526004    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/461521b6-3031-46d9-8e54-765c46db9983-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2\" (UID: \"461521b6-3031-46d9-8e54-765c46db9983\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2"
Oct 30 20:15:00.527021 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.526998    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z"]
Oct 30 20:15:00.527177 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.527015    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5\" (UID: \"9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5"
Oct 30 20:15:00.527177 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.526037    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8\" (UID: \"a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8"
Oct 30 20:15:00.527177 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.527166    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbf4r\" (UniqueName: \"kubernetes.io/projected/17e42313-2fdd-44dc-9f24-07c9f8283813-kube-api-access-tbf4r\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t\" (UID: \"17e42313-2fdd-44dc-9f24-07c9f8283813\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t"
Oct 30 20:15:00.527360 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.527199    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62ssf\" (UniqueName: \"kubernetes.io/projected/a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f-kube-api-access-62ssf\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8\" (UID: \"a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8"
Oct 30 20:15:00.527360 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.527223    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e42313-2fdd-44dc-9f24-07c9f8283813-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t\" (UID: \"17e42313-2fdd-44dc-9f24-07c9f8283813\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t"
Oct 30 20:15:00.528010 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.527987    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/461521b6-3031-46d9-8e54-765c46db9983-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2\" (UID: \"461521b6-3031-46d9-8e54-765c46db9983\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2"
Oct 30 20:15:00.550639 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.550581    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xc6k\" (UniqueName: \"kubernetes.io/projected/9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b-kube-api-access-6xc6k\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5\" (UID: \"9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5"
Oct 30 20:15:00.552173 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.552148    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4szcq\" (UniqueName: \"kubernetes.io/projected/461521b6-3031-46d9-8e54-765c46db9983-kube-api-access-4szcq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2\" (UID: \"461521b6-3031-46d9-8e54-765c46db9983\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2"
Oct 30 20:15:00.601887 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.601860    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28"
Oct 30 20:15:00.628143 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.628114    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4mdw\" (UniqueName: \"kubernetes.io/projected/6e28b0af-fad5-44bb-b949-08d292e74c97-kube-api-access-d4mdw\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z\" (UID: \"6e28b0af-fad5-44bb-b949-08d292e74c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z"
Oct 30 20:15:00.628330 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.628296    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e28b0af-fad5-44bb-b949-08d292e74c97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z\" (UID: \"6e28b0af-fad5-44bb-b949-08d292e74c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z"
Oct 30 20:15:00.628467 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.628352    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4z97\" (UniqueName: \"kubernetes.io/projected/75e2878b-a4f8-4180-a5c2-cb169475f2d7-kube-api-access-b4z97\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7\" (UID: \"75e2878b-a4f8-4180-a5c2-cb169475f2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7"
Oct 30 20:15:00.628467 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.628438    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380956ae-205c-4814-aa2d-68aae67dd2cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-7pmtv\" (UID: \"380956ae-205c-4814-aa2d-68aae67dd2cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv"
Oct 30 20:15:00.628586 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.628478    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59l2f\" (UniqueName: \"kubernetes.io/projected/380956ae-205c-4814-aa2d-68aae67dd2cc-kube-api-access-59l2f\") pod \"maas-api-key-cleanup-29889855-7pmtv\" (UID: \"380956ae-205c-4814-aa2d-68aae67dd2cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv"
Oct 30 20:15:00.628586 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.628523    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e2878b-a4f8-4180-a5c2-cb169475f2d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7\" (UID: \"75e2878b-a4f8-4180-a5c2-cb169475f2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7"
Oct 30 20:15:00.628586 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.628580    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8\" (UID: \"a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8"
Oct 30 20:15:00.628716 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.628606    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbf4r\" (UniqueName: \"kubernetes.io/projected/17e42313-2fdd-44dc-9f24-07c9f8283813-kube-api-access-tbf4r\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t\" (UID: \"17e42313-2fdd-44dc-9f24-07c9f8283813\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t"
Oct 30 20:15:00.628716 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.628634    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62ssf\" (UniqueName: \"kubernetes.io/projected/a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f-kube-api-access-62ssf\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8\" (UID: \"a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8"
Oct 30 20:15:00.628716 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.628659    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e42313-2fdd-44dc-9f24-07c9f8283813-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t\" (UID: \"17e42313-2fdd-44dc-9f24-07c9f8283813\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t"
Oct 30 20:15:00.629243 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.629222    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e2878b-a4f8-4180-a5c2-cb169475f2d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7\" (UID: \"75e2878b-a4f8-4180-a5c2-cb169475f2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7"
Oct 30 20:15:00.629304 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.629286    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e42313-2fdd-44dc-9f24-07c9f8283813-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t\" (UID: \"17e42313-2fdd-44dc-9f24-07c9f8283813\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t"
Oct 30 20:15:00.629703 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.629686    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380956ae-205c-4814-aa2d-68aae67dd2cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-7pmtv\" (UID: \"380956ae-205c-4814-aa2d-68aae67dd2cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv"
Oct 30 20:15:00.629759 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.629717    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8\" (UID: \"a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8"
Oct 30 20:15:00.645997 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.645970    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4z97\" (UniqueName: \"kubernetes.io/projected/75e2878b-a4f8-4180-a5c2-cb169475f2d7-kube-api-access-b4z97\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7\" (UID: \"75e2878b-a4f8-4180-a5c2-cb169475f2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7"
Oct 30 20:15:00.646170 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.646114    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59l2f\" (UniqueName: \"kubernetes.io/projected/380956ae-205c-4814-aa2d-68aae67dd2cc-kube-api-access-59l2f\") pod \"maas-api-key-cleanup-29889855-7pmtv\" (UID: \"380956ae-205c-4814-aa2d-68aae67dd2cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv"
Oct 30 20:15:00.646682 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.646655    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbf4r\" (UniqueName: \"kubernetes.io/projected/17e42313-2fdd-44dc-9f24-07c9f8283813-kube-api-access-tbf4r\") pod \"maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t\" (UID: \"17e42313-2fdd-44dc-9f24-07c9f8283813\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t"
Oct 30 20:15:00.647013 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.646997    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62ssf\" (UniqueName: \"kubernetes.io/projected/a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f-kube-api-access-62ssf\") pod \"maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8\" (UID: \"a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8"
Oct 30 20:15:00.655933 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.655907    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2"
Oct 30 20:15:00.664331 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.664304    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5"
Oct 30 20:15:00.711535 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.711327    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8"
Oct 30 20:15:00.730161 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.730129    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e28b0af-fad5-44bb-b949-08d292e74c97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z\" (UID: \"6e28b0af-fad5-44bb-b949-08d292e74c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z"
Oct 30 20:15:00.730331 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.730314    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4mdw\" (UniqueName: \"kubernetes.io/projected/6e28b0af-fad5-44bb-b949-08d292e74c97-kube-api-access-d4mdw\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z\" (UID: \"6e28b0af-fad5-44bb-b949-08d292e74c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z"
Oct 30 20:15:00.732258 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.731425    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e28b0af-fad5-44bb-b949-08d292e74c97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z\" (UID: \"6e28b0af-fad5-44bb-b949-08d292e74c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z"
Oct 30 20:15:00.732712 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.732535    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv"
Oct 30 20:15:00.761326 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.761261    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4mdw\" (UniqueName: \"kubernetes.io/projected/6e28b0af-fad5-44bb-b949-08d292e74c97-kube-api-access-d4mdw\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z\" (UID: \"6e28b0af-fad5-44bb-b949-08d292e74c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z"
Oct 30 20:15:00.793576 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.793520    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t"
Oct 30 20:15:00.809249 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.801448    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7"
Oct 30 20:15:00.822523 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.819970    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z"
Oct 30 20:15:00.918454 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.918415    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28"]
Oct 30 20:15:00.981304 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.981226    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5"]
Oct 30 20:15:00.983541 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:15:00.983487    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cbc3db5_b092_4b2a_8c94_baf0afa8ce5b.slice/crio-d621390c6842abffda1d3253c3afad906a49005a95f4bac379a0a3cf6564cb47 WatchSource:0}: Error finding container d621390c6842abffda1d3253c3afad906a49005a95f4bac379a0a3cf6564cb47: Status 404 returned error can't find the container with id d621390c6842abffda1d3253c3afad906a49005a95f4bac379a0a3cf6564cb47
Oct 30 20:15:00.987412 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:15:00.987381    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461521b6_3031_46d9_8e54_765c46db9983.slice/crio-03d64cc8f1928c8efa0127fe41d1b457efabf55aa4f49750f4ed9d3e654a723f WatchSource:0}: Error finding container 03d64cc8f1928c8efa0127fe41d1b457efabf55aa4f49750f4ed9d3e654a723f: Status 404 returned error can't find the container with id 03d64cc8f1928c8efa0127fe41d1b457efabf55aa4f49750f4ed9d3e654a723f
Oct 30 20:15:00.995691 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:00.995651    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2"]
Oct 30 20:15:01.043670 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:01.043636    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8"]
Oct 30 20:15:01.044737 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:15:01.044697    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a3b6f0_3fb7_4eb3_aba4_f2966f80bb7f.slice/crio-aae6336612aa355cfe375e50aaab2db7bc649efefbb6276b0941d57c952c2dcb WatchSource:0}: Error finding container aae6336612aa355cfe375e50aaab2db7bc649efefbb6276b0941d57c952c2dcb: Status 404 returned error can't find the container with id aae6336612aa355cfe375e50aaab2db7bc649efefbb6276b0941d57c952c2dcb
Oct 30 20:15:01.067227 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:15:01.067181    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380956ae_205c_4814_aa2d_68aae67dd2cc.slice/crio-0ce340e7d145a8a7abec91cd7f5bbc04e2cabd69e023df4e64f1ceb856635902 WatchSource:0}: Error finding container 0ce340e7d145a8a7abec91cd7f5bbc04e2cabd69e023df4e64f1ceb856635902: Status 404 returned error can't find the container with id 0ce340e7d145a8a7abec91cd7f5bbc04e2cabd69e023df4e64f1ceb856635902
Oct 30 20:15:01.067706 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:01.067683    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv"]
Oct 30 20:15:01.137411 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:01.137376    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z"]
Oct 30 20:15:01.164287 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:01.164264    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7"]
Oct 30 20:15:01.167366 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:01.167342    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t"]
Oct 30 20:15:01.176102 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:15:01.176074    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e28b0af_fad5_44bb_b949_08d292e74c97.slice/crio-68e9a577bee327c2db9391b46913cb96ca162ee6c4b4b988e4b8178111a26d7d WatchSource:0}: Error finding container 68e9a577bee327c2db9391b46913cb96ca162ee6c4b4b988e4b8178111a26d7d: Status 404 returned error can't find the container with id 68e9a577bee327c2db9391b46913cb96ca162ee6c4b4b988e4b8178111a26d7d
Oct 30 20:15:01.177277 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:15:01.177255    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e2878b_a4f8_4180_a5c2_cb169475f2d7.slice/crio-6d30d4ba06b12d39d232dd6d3701edd8555bbc1068928739005d0259c9587014 WatchSource:0}: Error finding container 6d30d4ba06b12d39d232dd6d3701edd8555bbc1068928739005d0259c9587014: Status 404 returned error can't find the container with id 6d30d4ba06b12d39d232dd6d3701edd8555bbc1068928739005d0259c9587014
Oct 30 20:15:01.178314 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:15:01.178289    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e42313_2fdd_44dc_9f24_07c9f8283813.slice/crio-396f1df24c15ef6994b8045267199b974bb75d183e50afa1d5252c07b6649de8 WatchSource:0}: Error finding container 396f1df24c15ef6994b8045267199b974bb75d183e50afa1d5252c07b6649de8: Status 404 returned error can't find the container with id 396f1df24c15ef6994b8045267199b974bb75d183e50afa1d5252c07b6649de8
Oct 30 20:15:01.440975 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:01.440935    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8" event={"ID":"a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f","Type":"ContainerStarted","Data":"aae6336612aa355cfe375e50aaab2db7bc649efefbb6276b0941d57c952c2dcb"}
Oct 30 20:15:01.442839 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:01.442807    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z" event={"ID":"6e28b0af-fad5-44bb-b949-08d292e74c97","Type":"ContainerStarted","Data":"68e9a577bee327c2db9391b46913cb96ca162ee6c4b4b988e4b8178111a26d7d"}
Oct 30 20:15:01.445449 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:01.445420    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t" event={"ID":"17e42313-2fdd-44dc-9f24-07c9f8283813","Type":"ContainerStarted","Data":"396f1df24c15ef6994b8045267199b974bb75d183e50afa1d5252c07b6649de8"}
Oct 30 20:15:01.446940 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:01.446884    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28" event={"ID":"4ca670d4-cee3-4066-80e6-8b3a5ca7783b","Type":"ContainerStarted","Data":"2ff7c3c2fff669bcad40c9dfefe2d203bcea14de24004984d76f8eb748bf8c22"}
Oct 30 20:15:01.448341 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:01.448311    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7" event={"ID":"75e2878b-a4f8-4180-a5c2-cb169475f2d7","Type":"ContainerStarted","Data":"6d30d4ba06b12d39d232dd6d3701edd8555bbc1068928739005d0259c9587014"}
Oct 30 20:15:01.449645 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:01.449625    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv" event={"ID":"380956ae-205c-4814-aa2d-68aae67dd2cc","Type":"ContainerStarted","Data":"0ce340e7d145a8a7abec91cd7f5bbc04e2cabd69e023df4e64f1ceb856635902"}
Oct 30 20:15:01.450998 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:01.450975    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2" event={"ID":"461521b6-3031-46d9-8e54-765c46db9983","Type":"ContainerStarted","Data":"03d64cc8f1928c8efa0127fe41d1b457efabf55aa4f49750f4ed9d3e654a723f"}
Oct 30 20:15:01.452229 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:01.452202    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5" event={"ID":"9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b","Type":"ContainerStarted","Data":"d621390c6842abffda1d3253c3afad906a49005a95f4bac379a0a3cf6564cb47"}
Oct 30 20:15:02.468557 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.468523    6030 generic.go:356] "Generic (PLEG): container finished" podID="461521b6-3031-46d9-8e54-765c46db9983" containerID="46d134ebb0bae455f404797703a38a10315c4c88569fa8460744d505d814294c" exitCode=60
Oct 30 20:15:02.468963 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.468617    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2" event={"ID":"461521b6-3031-46d9-8e54-765c46db9983","Type":"ContainerDied","Data":"46d134ebb0bae455f404797703a38a10315c4c88569fa8460744d505d814294c"}
Oct 30 20:15:02.468963 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.468856    6030 scope.go:122] "RemoveContainer" containerID="46d134ebb0bae455f404797703a38a10315c4c88569fa8460744d505d814294c"
Oct 30 20:15:02.470963 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.470939    6030 generic.go:356] "Generic (PLEG): container finished" podID="9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b" containerID="def438394f1d4c315980f8156cd916a33c6b076a15924b8cb53a549e262ce269" exitCode=60
Oct 30 20:15:02.471104 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.471019    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5" event={"ID":"9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b","Type":"ContainerDied","Data":"def438394f1d4c315980f8156cd916a33c6b076a15924b8cb53a549e262ce269"}
Oct 30 20:15:02.471268 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.471246    6030 scope.go:122] "RemoveContainer" containerID="def438394f1d4c315980f8156cd916a33c6b076a15924b8cb53a549e262ce269"
Oct 30 20:15:02.473306 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.473283    6030 generic.go:356] "Generic (PLEG): container finished" podID="a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f" containerID="703934fcd855064b04e5610e1171cdccb644f188231b3b71504a3bf276068845" exitCode=60
Oct 30 20:15:02.473403 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.473378    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8" event={"ID":"a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f","Type":"ContainerDied","Data":"703934fcd855064b04e5610e1171cdccb644f188231b3b71504a3bf276068845"}
Oct 30 20:15:02.474394 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.474376    6030 scope.go:122] "RemoveContainer" containerID="703934fcd855064b04e5610e1171cdccb644f188231b3b71504a3bf276068845"
Oct 30 20:15:02.476161 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.476136    6030 generic.go:356] "Generic (PLEG): container finished" podID="6e28b0af-fad5-44bb-b949-08d292e74c97" containerID="a5c780b9926e533379584f2ccc0c0fcf1b0d5fa79f932765004d4c0821bdeb83" exitCode=60
Oct 30 20:15:02.476257 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.476212    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z" event={"ID":"6e28b0af-fad5-44bb-b949-08d292e74c97","Type":"ContainerDied","Data":"a5c780b9926e533379584f2ccc0c0fcf1b0d5fa79f932765004d4c0821bdeb83"}
Oct 30 20:15:02.476422 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.476407    6030 scope.go:122] "RemoveContainer" containerID="a5c780b9926e533379584f2ccc0c0fcf1b0d5fa79f932765004d4c0821bdeb83"
Oct 30 20:15:02.477976 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.477945    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t" event={"ID":"17e42313-2fdd-44dc-9f24-07c9f8283813","Type":"ContainerStarted","Data":"a1c24093beb900719f1c0e5b43857caf04eb6c50b91a253979258f550e2babab"}
Oct 30 20:15:02.479647 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.479626    6030 generic.go:356] "Generic (PLEG): container finished" podID="4ca670d4-cee3-4066-80e6-8b3a5ca7783b" containerID="ef1c9f4ad1283283e89a64c2d22736bd9865f375dbc268769fabc49d932965b2" exitCode=60
Oct 30 20:15:02.479647 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.479653    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28" event={"ID":"4ca670d4-cee3-4066-80e6-8b3a5ca7783b","Type":"ContainerDied","Data":"ef1c9f4ad1283283e89a64c2d22736bd9865f375dbc268769fabc49d932965b2"}
Oct 30 20:15:02.479943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.479925    6030 scope.go:122] "RemoveContainer" containerID="ef1c9f4ad1283283e89a64c2d22736bd9865f375dbc268769fabc49d932965b2"
Oct 30 20:15:02.481430 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.481399    6030 generic.go:356] "Generic (PLEG): container finished" podID="75e2878b-a4f8-4180-a5c2-cb169475f2d7" containerID="d4e123e209aeff8ed9e0e270b5a3c11495eb9be9e7534564d72794cd0f6f6e69" exitCode=60
Oct 30 20:15:02.481534 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.481474    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7" event={"ID":"75e2878b-a4f8-4180-a5c2-cb169475f2d7","Type":"ContainerDied","Data":"d4e123e209aeff8ed9e0e270b5a3c11495eb9be9e7534564d72794cd0f6f6e69"}
Oct 30 20:15:02.481656 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.481633    6030 scope.go:122] "RemoveContainer" containerID="d4e123e209aeff8ed9e0e270b5a3c11495eb9be9e7534564d72794cd0f6f6e69"
Oct 30 20:15:02.483574 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.483554    6030 generic.go:356] "Generic (PLEG): container finished" podID="380956ae-205c-4814-aa2d-68aae67dd2cc" containerID="397bdd88384595376f898a966477d5e40a91c3ee1b481f4a9df25a33daad26ab" exitCode=60
Oct 30 20:15:02.483687 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.483621    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv" event={"ID":"380956ae-205c-4814-aa2d-68aae67dd2cc","Type":"ContainerDied","Data":"397bdd88384595376f898a966477d5e40a91c3ee1b481f4a9df25a33daad26ab"}
Oct 30 20:15:02.483824 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.483808    6030 scope.go:122] "RemoveContainer" containerID="397bdd88384595376f898a966477d5e40a91c3ee1b481f4a9df25a33daad26ab"
Oct 30 20:15:02.545969 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:02.545927    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t" podStartSLOduration=2.545912965 podStartE2EDuration="2.545912965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.539205055 +0000 UTC m=+2581.454811912" watchObservedRunningTime="2026-10-30 20:15:02.545912965 +0000 UTC m=+2581.461519820"
Oct 30 20:15:03.492358 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:03.492322    6030 generic.go:356] "Generic (PLEG): container finished" podID="17e42313-2fdd-44dc-9f24-07c9f8283813" containerID="a1c24093beb900719f1c0e5b43857caf04eb6c50b91a253979258f550e2babab" exitCode=60
Oct 30 20:15:03.492815 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:03.492403    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t" event={"ID":"17e42313-2fdd-44dc-9f24-07c9f8283813","Type":"ContainerDied","Data":"a1c24093beb900719f1c0e5b43857caf04eb6c50b91a253979258f550e2babab"}
Oct 30 20:15:03.492815 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:03.492721    6030 scope.go:122] "RemoveContainer" containerID="a1c24093beb900719f1c0e5b43857caf04eb6c50b91a253979258f550e2babab"
Oct 30 20:15:03.494623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:03.494594    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28" event={"ID":"4ca670d4-cee3-4066-80e6-8b3a5ca7783b","Type":"ContainerStarted","Data":"dc022977b23972a10ec804433130eda0bfc19ce11e47d58e8fd2ffcf3c70259a"}
Oct 30 20:15:03.498738 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:03.498687    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7" event={"ID":"75e2878b-a4f8-4180-a5c2-cb169475f2d7","Type":"ContainerStarted","Data":"1c6dc9ec6a2a6aeb403e072f27aa4718ad8bc9fdec96a34463cd758ac804f334"}
Oct 30 20:15:03.501316 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:03.501295    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv" event={"ID":"380956ae-205c-4814-aa2d-68aae67dd2cc","Type":"ContainerStarted","Data":"b675c166f1153715ef35f79f2d86802c9a22c3b173bd1988a9381b26d631c570"}
Oct 30 20:15:03.503313 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:03.503287    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2" event={"ID":"461521b6-3031-46d9-8e54-765c46db9983","Type":"ContainerStarted","Data":"04c833f81579921a113f9b1f99150a65c863e7e7d79d235072dde425fa405f33"}
Oct 30 20:15:03.505316 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:03.505295    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5" event={"ID":"9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b","Type":"ContainerStarted","Data":"1b19e3ebd9bebdd38b060ad6d5ff8008be5c0d3308fb0a7523332901a2eec98a"}
Oct 30 20:15:03.507329 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:03.507307    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8" event={"ID":"a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f","Type":"ContainerStarted","Data":"abdc9b4fd92a095d142e7f4a1a3edaf27ec3650f26f41cab72ec31fcca42cb2f"}
Oct 30 20:15:03.509098 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:03.509075    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z" event={"ID":"6e28b0af-fad5-44bb-b949-08d292e74c97","Type":"ContainerStarted","Data":"b243bd97c978d933003a160e9cd62aa0fc1b0b53d6ad05bc4bedea707056e6ba"}
Oct 30 20:15:03.538873 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:03.538784    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z" podStartSLOduration=3.538771121 podStartE2EDuration="3.538771121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:03.535790939 +0000 UTC m=+2582.451397796" watchObservedRunningTime="2026-10-30 20:15:03.538771121 +0000 UTC m=+2582.454377974"
Oct 30 20:15:03.557888 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:03.557831    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5" podStartSLOduration=3.557817902 podStartE2EDuration="3.557817902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:03.555514502 +0000 UTC m=+2582.471121359" watchObservedRunningTime="2026-10-30 20:15:03.557817902 +0000 UTC m=+2582.473424760"
Oct 30 20:15:03.581162 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:03.581104    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2" podStartSLOduration=3.581085518 podStartE2EDuration="3.581085518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:03.578190985 +0000 UTC m=+2582.493797844" watchObservedRunningTime="2026-10-30 20:15:03.581085518 +0000 UTC m=+2582.496692375"
Oct 30 20:15:03.605437 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:03.605384    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7" podStartSLOduration=3.605370673 podStartE2EDuration="3.605370673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:03.599797233 +0000 UTC m=+2582.515404090" watchObservedRunningTime="2026-10-30 20:15:03.605370673 +0000 UTC m=+2582.520977532"
Oct 30 20:15:03.621546 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:03.621489    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28" podStartSLOduration=3.621475105 podStartE2EDuration="3.621475105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:03.619363529 +0000 UTC m=+2582.534970385" watchObservedRunningTime="2026-10-30 20:15:03.621475105 +0000 UTC m=+2582.537081960"
Oct 30 20:15:03.646915 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:03.646854    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8" podStartSLOduration=3.646836269 podStartE2EDuration="3.646836269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:03.643824329 +0000 UTC m=+2582.559431185" watchObservedRunningTime="2026-10-30 20:15:03.646836269 +0000 UTC m=+2582.562443130"
Oct 30 20:15:03.689710 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:03.689663    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv" podStartSLOduration=3.689647061 podStartE2EDuration="3.689647061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:03.671765408 +0000 UTC m=+2582.587372269" watchObservedRunningTime="2026-10-30 20:15:03.689647061 +0000 UTC m=+2582.605253916"
Oct 30 20:15:04.289003 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:04.288970    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:15:04.528823 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.528791    6030 generic.go:356] "Generic (PLEG): container finished" podID="17e42313-2fdd-44dc-9f24-07c9f8283813" containerID="6d3682975c0b387cd8322063c2287c5a708b54282fc7afdd9a53814641c3a379" exitCode=60
Oct 30 20:15:04.529290 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.528878    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t" event={"ID":"17e42313-2fdd-44dc-9f24-07c9f8283813","Type":"ContainerDied","Data":"6d3682975c0b387cd8322063c2287c5a708b54282fc7afdd9a53814641c3a379"}
Oct 30 20:15:04.529290 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.528956    6030 scope.go:122] "RemoveContainer" containerID="a1c24093beb900719f1c0e5b43857caf04eb6c50b91a253979258f550e2babab"
Oct 30 20:15:04.529457 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.529406    6030 scope.go:122] "RemoveContainer" containerID="6d3682975c0b387cd8322063c2287c5a708b54282fc7afdd9a53814641c3a379"
Oct 30 20:15:04.529762 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:04.529733    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t_odh-ai-gateway-infra(17e42313-2fdd-44dc-9f24-07c9f8283813)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t" podUID="17e42313-2fdd-44dc-9f24-07c9f8283813"
Oct 30 20:15:04.531144 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.531119    6030 generic.go:356] "Generic (PLEG): container finished" podID="4ca670d4-cee3-4066-80e6-8b3a5ca7783b" containerID="dc022977b23972a10ec804433130eda0bfc19ce11e47d58e8fd2ffcf3c70259a" exitCode=60
Oct 30 20:15:04.531239 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.531162    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28" event={"ID":"4ca670d4-cee3-4066-80e6-8b3a5ca7783b","Type":"ContainerDied","Data":"dc022977b23972a10ec804433130eda0bfc19ce11e47d58e8fd2ffcf3c70259a"}
Oct 30 20:15:04.531481 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.531463    6030 scope.go:122] "RemoveContainer" containerID="dc022977b23972a10ec804433130eda0bfc19ce11e47d58e8fd2ffcf3c70259a"
Oct 30 20:15:04.531767 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:04.531750    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28_odh-ai-gateway-infra(4ca670d4-cee3-4066-80e6-8b3a5ca7783b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28" podUID="4ca670d4-cee3-4066-80e6-8b3a5ca7783b"
Oct 30 20:15:04.533273 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.533251    6030 generic.go:356] "Generic (PLEG): container finished" podID="75e2878b-a4f8-4180-a5c2-cb169475f2d7" containerID="1c6dc9ec6a2a6aeb403e072f27aa4718ad8bc9fdec96a34463cd758ac804f334" exitCode=60
Oct 30 20:15:04.533390 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.533328    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7" event={"ID":"75e2878b-a4f8-4180-a5c2-cb169475f2d7","Type":"ContainerDied","Data":"1c6dc9ec6a2a6aeb403e072f27aa4718ad8bc9fdec96a34463cd758ac804f334"}
Oct 30 20:15:04.533677 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.533635    6030 scope.go:122] "RemoveContainer" containerID="1c6dc9ec6a2a6aeb403e072f27aa4718ad8bc9fdec96a34463cd758ac804f334"
Oct 30 20:15:04.533978 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:04.533954    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7_odh-ai-gateway-infra(75e2878b-a4f8-4180-a5c2-cb169475f2d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7" podUID="75e2878b-a4f8-4180-a5c2-cb169475f2d7"
Oct 30 20:15:04.535366 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.535347    6030 generic.go:356] "Generic (PLEG): container finished" podID="380956ae-205c-4814-aa2d-68aae67dd2cc" containerID="b675c166f1153715ef35f79f2d86802c9a22c3b173bd1988a9381b26d631c570" exitCode=60
Oct 30 20:15:04.535488 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.535465    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv" event={"ID":"380956ae-205c-4814-aa2d-68aae67dd2cc","Type":"ContainerDied","Data":"b675c166f1153715ef35f79f2d86802c9a22c3b173bd1988a9381b26d631c570"}
Oct 30 20:15:04.535631 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.535609    6030 scope.go:122] "RemoveContainer" containerID="b675c166f1153715ef35f79f2d86802c9a22c3b173bd1988a9381b26d631c570"
Oct 30 20:15:04.535908 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:04.535884    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-7pmtv_odh-ai-gateway-infra(380956ae-205c-4814-aa2d-68aae67dd2cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv" podUID="380956ae-205c-4814-aa2d-68aae67dd2cc"
Oct 30 20:15:04.537386 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.537364    6030 generic.go:356] "Generic (PLEG): container finished" podID="461521b6-3031-46d9-8e54-765c46db9983" containerID="04c833f81579921a113f9b1f99150a65c863e7e7d79d235072dde425fa405f33" exitCode=60
Oct 30 20:15:04.537497 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.537450    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2" event={"ID":"461521b6-3031-46d9-8e54-765c46db9983","Type":"ContainerDied","Data":"04c833f81579921a113f9b1f99150a65c863e7e7d79d235072dde425fa405f33"}
Oct 30 20:15:04.537660 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.537643    6030 scope.go:122] "RemoveContainer" containerID="04c833f81579921a113f9b1f99150a65c863e7e7d79d235072dde425fa405f33"
Oct 30 20:15:04.537935 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:04.537918    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2_odh-ai-gateway-infra(461521b6-3031-46d9-8e54-765c46db9983)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2" podUID="461521b6-3031-46d9-8e54-765c46db9983"
Oct 30 20:15:04.539321 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.539303    6030 generic.go:356] "Generic (PLEG): container finished" podID="9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b" containerID="1b19e3ebd9bebdd38b060ad6d5ff8008be5c0d3308fb0a7523332901a2eec98a" exitCode=60
Oct 30 20:15:04.539411 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.539370    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5" event={"ID":"9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b","Type":"ContainerDied","Data":"1b19e3ebd9bebdd38b060ad6d5ff8008be5c0d3308fb0a7523332901a2eec98a"}
Oct 30 20:15:04.539582 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.539569    6030 scope.go:122] "RemoveContainer" containerID="1b19e3ebd9bebdd38b060ad6d5ff8008be5c0d3308fb0a7523332901a2eec98a"
Oct 30 20:15:04.539792 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:04.539776    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5_odh-ai-gateway-infra(9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5" podUID="9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b"
Oct 30 20:15:04.541472 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.541453    6030 generic.go:356] "Generic (PLEG): container finished" podID="a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f" containerID="abdc9b4fd92a095d142e7f4a1a3edaf27ec3650f26f41cab72ec31fcca42cb2f" exitCode=60
Oct 30 20:15:04.541590 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.541481    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8" event={"ID":"a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f","Type":"ContainerDied","Data":"abdc9b4fd92a095d142e7f4a1a3edaf27ec3650f26f41cab72ec31fcca42cb2f"}
Oct 30 20:15:04.541913 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.541887    6030 scope.go:122] "RemoveContainer" containerID="abdc9b4fd92a095d142e7f4a1a3edaf27ec3650f26f41cab72ec31fcca42cb2f"
Oct 30 20:15:04.542194 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:04.542163    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8_odh-ai-gateway-infra(a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8" podUID="a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f"
Oct 30 20:15:04.543735 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.543636    6030 generic.go:356] "Generic (PLEG): container finished" podID="6e28b0af-fad5-44bb-b949-08d292e74c97" containerID="b243bd97c978d933003a160e9cd62aa0fc1b0b53d6ad05bc4bedea707056e6ba" exitCode=60
Oct 30 20:15:04.543735 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.543692    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z" event={"ID":"6e28b0af-fad5-44bb-b949-08d292e74c97","Type":"ContainerDied","Data":"b243bd97c978d933003a160e9cd62aa0fc1b0b53d6ad05bc4bedea707056e6ba"}
Oct 30 20:15:04.543735 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.543717    6030 scope.go:122] "RemoveContainer" containerID="ef1c9f4ad1283283e89a64c2d22736bd9865f375dbc268769fabc49d932965b2"
Oct 30 20:15:04.543954 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.543936    6030 scope.go:122] "RemoveContainer" containerID="b243bd97c978d933003a160e9cd62aa0fc1b0b53d6ad05bc4bedea707056e6ba"
Oct 30 20:15:04.544304 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:04.544262    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z_odh-ai-gateway-infra(6e28b0af-fad5-44bb-b949-08d292e74c97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z" podUID="6e28b0af-fad5-44bb-b949-08d292e74c97"
Oct 30 20:15:04.559170 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.559150    6030 scope.go:122] "RemoveContainer" containerID="d4e123e209aeff8ed9e0e270b5a3c11495eb9be9e7534564d72794cd0f6f6e69"
Oct 30 20:15:04.572784 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.572767    6030 scope.go:122] "RemoveContainer" containerID="397bdd88384595376f898a966477d5e40a91c3ee1b481f4a9df25a33daad26ab"
Oct 30 20:15:04.622415 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.622392    6030 scope.go:122] "RemoveContainer" containerID="46d134ebb0bae455f404797703a38a10315c4c88569fa8460744d505d814294c"
Oct 30 20:15:04.637113 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.637093    6030 scope.go:122] "RemoveContainer" containerID="def438394f1d4c315980f8156cd916a33c6b076a15924b8cb53a549e262ce269"
Oct 30 20:15:04.657164 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.657123    6030 scope.go:122] "RemoveContainer" containerID="703934fcd855064b04e5610e1171cdccb644f188231b3b71504a3bf276068845"
Oct 30 20:15:04.677372 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:04.677285    6030 scope.go:122] "RemoveContainer" containerID="a5c780b9926e533379584f2ccc0c0fcf1b0d5fa79f932765004d4c0821bdeb83"
Oct 30 20:15:05.383242 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:05.383209    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"
Oct 30 20:15:05.383417 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:05.383281    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"
Oct 30 20:15:05.553564 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:05.553535    6030 scope.go:122] "RemoveContainer" containerID="04c833f81579921a113f9b1f99150a65c863e7e7d79d235072dde425fa405f33"
Oct 30 20:15:05.554007 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:05.553985    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2_odh-ai-gateway-infra(461521b6-3031-46d9-8e54-765c46db9983)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2" podUID="461521b6-3031-46d9-8e54-765c46db9983"
Oct 30 20:15:05.555736 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:05.555707    6030 scope.go:122] "RemoveContainer" containerID="1b19e3ebd9bebdd38b060ad6d5ff8008be5c0d3308fb0a7523332901a2eec98a"
Oct 30 20:15:05.556144 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:05.556118    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5_odh-ai-gateway-infra(9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5" podUID="9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b"
Oct 30 20:15:05.557585 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:05.557562    6030 scope.go:122] "RemoveContainer" containerID="abdc9b4fd92a095d142e7f4a1a3edaf27ec3650f26f41cab72ec31fcca42cb2f"
Oct 30 20:15:05.557907 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:05.557885    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8_odh-ai-gateway-infra(a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8" podUID="a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f"
Oct 30 20:15:05.559306 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:05.559285    6030 scope.go:122] "RemoveContainer" containerID="b243bd97c978d933003a160e9cd62aa0fc1b0b53d6ad05bc4bedea707056e6ba"
Oct 30 20:15:05.559558 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:05.559537    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z_odh-ai-gateway-infra(6e28b0af-fad5-44bb-b949-08d292e74c97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z" podUID="6e28b0af-fad5-44bb-b949-08d292e74c97"
Oct 30 20:15:05.561157 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:05.561138    6030 scope.go:122] "RemoveContainer" containerID="6d3682975c0b387cd8322063c2287c5a708b54282fc7afdd9a53814641c3a379"
Oct 30 20:15:05.561425 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:05.561406    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t_odh-ai-gateway-infra(17e42313-2fdd-44dc-9f24-07c9f8283813)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t" podUID="17e42313-2fdd-44dc-9f24-07c9f8283813"
Oct 30 20:15:05.562883 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:05.562863    6030 scope.go:122] "RemoveContainer" containerID="dc022977b23972a10ec804433130eda0bfc19ce11e47d58e8fd2ffcf3c70259a"
Oct 30 20:15:05.563179 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:05.563155    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28_odh-ai-gateway-infra(4ca670d4-cee3-4066-80e6-8b3a5ca7783b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28" podUID="4ca670d4-cee3-4066-80e6-8b3a5ca7783b"
Oct 30 20:15:05.564678 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:05.564588    6030 scope.go:122] "RemoveContainer" containerID="1c6dc9ec6a2a6aeb403e072f27aa4718ad8bc9fdec96a34463cd758ac804f334"
Oct 30 20:15:05.564853 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:05.564833    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7_odh-ai-gateway-infra(75e2878b-a4f8-4180-a5c2-cb169475f2d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7" podUID="75e2878b-a4f8-4180-a5c2-cb169475f2d7"
Oct 30 20:15:05.566162 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:05.566144    6030 scope.go:122] "RemoveContainer" containerID="b675c166f1153715ef35f79f2d86802c9a22c3b173bd1988a9381b26d631c570"
Oct 30 20:15:05.566379 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:05.566360    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-7pmtv_odh-ai-gateway-infra(380956ae-205c-4814-aa2d-68aae67dd2cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv" podUID="380956ae-205c-4814-aa2d-68aae67dd2cc"
Oct 30 20:15:11.944229 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:11.944186    6030 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-clbw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:15:11.944812 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:11.944778    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-clbw6" podUID="948623f8-d637-44f2-a460-43737e3384d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:15:12.829643 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:12.829611    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"]
Oct 30 20:15:12.836444 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:12.836421    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:12.845456 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:12.843301    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"agc-cred-8cb301183731\""
Oct 30 20:15:12.848024 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:12.848003    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"]
Oct 30 20:15:12.917995 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:12.917946    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/674f52f4-ad53-45f5-90ae-93ac0848ee23-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z\" (UID: \"674f52f4-ad53-45f5-90ae-93ac0848ee23\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:12.918308 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:12.918103    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/674f52f4-ad53-45f5-90ae-93ac0848ee23-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z\" (UID: \"674f52f4-ad53-45f5-90ae-93ac0848ee23\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:12.918308 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:12.918274    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl8qh\" (UniqueName: \"kubernetes.io/projected/674f52f4-ad53-45f5-90ae-93ac0848ee23-kube-api-access-wl8qh\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z\" (UID: \"674f52f4-ad53-45f5-90ae-93ac0848ee23\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:12.918528 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:12.918324    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/674f52f4-ad53-45f5-90ae-93ac0848ee23-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z\" (UID: \"674f52f4-ad53-45f5-90ae-93ac0848ee23\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:13.019977 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:13.019934    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl8qh\" (UniqueName: \"kubernetes.io/projected/674f52f4-ad53-45f5-90ae-93ac0848ee23-kube-api-access-wl8qh\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z\" (UID: \"674f52f4-ad53-45f5-90ae-93ac0848ee23\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:13.020487 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:13.019999    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/674f52f4-ad53-45f5-90ae-93ac0848ee23-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z\" (UID: \"674f52f4-ad53-45f5-90ae-93ac0848ee23\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:13.020487 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:13.020162    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/674f52f4-ad53-45f5-90ae-93ac0848ee23-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z\" (UID: \"674f52f4-ad53-45f5-90ae-93ac0848ee23\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:13.020487 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:13.020272    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/674f52f4-ad53-45f5-90ae-93ac0848ee23-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z\" (UID: \"674f52f4-ad53-45f5-90ae-93ac0848ee23\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:13.020765 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:13.020736    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/674f52f4-ad53-45f5-90ae-93ac0848ee23-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z\" (UID: \"674f52f4-ad53-45f5-90ae-93ac0848ee23\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:13.020967 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:13.020945    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/674f52f4-ad53-45f5-90ae-93ac0848ee23-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z\" (UID: \"674f52f4-ad53-45f5-90ae-93ac0848ee23\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:13.022926 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:13.022904    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/674f52f4-ad53-45f5-90ae-93ac0848ee23-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z\" (UID: \"674f52f4-ad53-45f5-90ae-93ac0848ee23\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:13.034484 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:13.034461    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl8qh\" (UniqueName: \"kubernetes.io/projected/674f52f4-ad53-45f5-90ae-93ac0848ee23-kube-api-access-wl8qh\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z\" (UID: \"674f52f4-ad53-45f5-90ae-93ac0848ee23\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:13.153982 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:13.153947    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:13.464340 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:13.464309    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"]
Oct 30 20:15:13.466659 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:15:13.466616    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674f52f4_ad53_45f5_90ae_93ac0848ee23.slice/crio-5780c0eecadc318118c79335effa8c5263b47fec078abb555f3161ebc3642712 WatchSource:0}: Error finding container 5780c0eecadc318118c79335effa8c5263b47fec078abb555f3161ebc3642712: Status 404 returned error can't find the container with id 5780c0eecadc318118c79335effa8c5263b47fec078abb555f3161ebc3642712
Oct 30 20:15:13.716517 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:13.716431    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z" event={"ID":"674f52f4-ad53-45f5-90ae-93ac0848ee23","Type":"ContainerStarted","Data":"4ff0eccd6e9663c1ad996a8807a3119f957cb60ab65a4a8c4cdb3d2ae1290b7c"}
Oct 30 20:15:13.716517 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:13.716486    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z" event={"ID":"674f52f4-ad53-45f5-90ae-93ac0848ee23","Type":"ContainerStarted","Data":"5780c0eecadc318118c79335effa8c5263b47fec078abb555f3161ebc3642712"}
Oct 30 20:15:13.716706 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:13.716524    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:13.744236 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:13.744171    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z" podStartSLOduration=1.7441570290000001 podStartE2EDuration="1.744157029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:13.743657716 +0000 UTC m=+2592.659264573" watchObservedRunningTime="2026-10-30 20:15:13.744157029 +0000 UTC m=+2592.659763884"
Oct 30 20:15:15.195066 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:15.194767    6030 scope.go:122] "RemoveContainer" containerID="abdc9b4fd92a095d142e7f4a1a3edaf27ec3650f26f41cab72ec31fcca42cb2f"
Oct 30 20:15:15.195561 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:15.195069    6030 scope.go:122] "RemoveContainer" containerID="b243bd97c978d933003a160e9cd62aa0fc1b0b53d6ad05bc4bedea707056e6ba"
Oct 30 20:15:15.195981 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:15.195775    6030 scope.go:122] "RemoveContainer" containerID="dc022977b23972a10ec804433130eda0bfc19ce11e47d58e8fd2ffcf3c70259a"
Oct 30 20:15:15.733450 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:15.733416    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8" event={"ID":"a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f","Type":"ContainerStarted","Data":"5a162e2f9f237be90ff562b51b73f9e97dbec6c89cebad01577785b68cc089e1"}
Oct 30 20:15:16.743162 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:16.743131    6030 generic.go:356] "Generic (PLEG): container finished" podID="a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f" containerID="5a162e2f9f237be90ff562b51b73f9e97dbec6c89cebad01577785b68cc089e1" exitCode=60
Oct 30 20:15:16.743626 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:16.743208    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8" event={"ID":"a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f","Type":"ContainerDied","Data":"5a162e2f9f237be90ff562b51b73f9e97dbec6c89cebad01577785b68cc089e1"}
Oct 30 20:15:16.743626 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:16.743272    6030 scope.go:122] "RemoveContainer" containerID="abdc9b4fd92a095d142e7f4a1a3edaf27ec3650f26f41cab72ec31fcca42cb2f"
Oct 30 20:15:16.743751 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:16.743664    6030 scope.go:122] "RemoveContainer" containerID="5a162e2f9f237be90ff562b51b73f9e97dbec6c89cebad01577785b68cc089e1"
Oct 30 20:15:16.743985 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:16.743960    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8_odh-ai-gateway-infra(a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8" podUID="a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f"
Oct 30 20:15:16.745645 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:16.745625    6030 generic.go:356] "Generic (PLEG): container finished" podID="6e28b0af-fad5-44bb-b949-08d292e74c97" containerID="d99120e68cbf34e58c8b90416f81d5a2e7eda74bd7bee24ed3a74a7648386a80" exitCode=60
Oct 30 20:15:16.745751 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:16.745687    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z" event={"ID":"6e28b0af-fad5-44bb-b949-08d292e74c97","Type":"ContainerDied","Data":"d99120e68cbf34e58c8b90416f81d5a2e7eda74bd7bee24ed3a74a7648386a80"}
Oct 30 20:15:16.746114 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:16.746097    6030 scope.go:122] "RemoveContainer" containerID="d99120e68cbf34e58c8b90416f81d5a2e7eda74bd7bee24ed3a74a7648386a80"
Oct 30 20:15:16.746382 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:16.746358    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z_odh-ai-gateway-infra(6e28b0af-fad5-44bb-b949-08d292e74c97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z" podUID="6e28b0af-fad5-44bb-b949-08d292e74c97"
Oct 30 20:15:16.747731 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:16.747709    6030 generic.go:356] "Generic (PLEG): container finished" podID="4ca670d4-cee3-4066-80e6-8b3a5ca7783b" containerID="2b8c552eea88627ac422aa4910ed2c6cdacefa3ad7d504b53a70a9c6b8c21855" exitCode=60
Oct 30 20:15:16.747818 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:16.747783    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28" event={"ID":"4ca670d4-cee3-4066-80e6-8b3a5ca7783b","Type":"ContainerDied","Data":"2b8c552eea88627ac422aa4910ed2c6cdacefa3ad7d504b53a70a9c6b8c21855"}
Oct 30 20:15:16.748336 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:16.748315    6030 scope.go:122] "RemoveContainer" containerID="2b8c552eea88627ac422aa4910ed2c6cdacefa3ad7d504b53a70a9c6b8c21855"
Oct 30 20:15:16.748637 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:16.748615    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28_odh-ai-gateway-infra(4ca670d4-cee3-4066-80e6-8b3a5ca7783b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28" podUID="4ca670d4-cee3-4066-80e6-8b3a5ca7783b"
Oct 30 20:15:16.756840 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:16.756820    6030 scope.go:122] "RemoveContainer" containerID="b243bd97c978d933003a160e9cd62aa0fc1b0b53d6ad05bc4bedea707056e6ba"
Oct 30 20:15:16.770897 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:16.770879    6030 scope.go:122] "RemoveContainer" containerID="dc022977b23972a10ec804433130eda0bfc19ce11e47d58e8fd2ffcf3c70259a"
Oct 30 20:15:16.782596 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:16.782575    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8"]
Oct 30 20:15:17.194189 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:17.194161    6030 scope.go:122] "RemoveContainer" containerID="04c833f81579921a113f9b1f99150a65c863e7e7d79d235072dde425fa405f33"
Oct 30 20:15:17.194441 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:17.194413    6030 scope.go:122] "RemoveContainer" containerID="1c6dc9ec6a2a6aeb403e072f27aa4718ad8bc9fdec96a34463cd758ac804f334"
Oct 30 20:15:17.195355 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:17.195184    6030 scope.go:122] "RemoveContainer" containerID="6d3682975c0b387cd8322063c2287c5a708b54282fc7afdd9a53814641c3a379"
Oct 30 20:15:17.195355 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:17.195287    6030 scope.go:122] "RemoveContainer" containerID="1b19e3ebd9bebdd38b060ad6d5ff8008be5c0d3308fb0a7523332901a2eec98a"
Oct 30 20:15:17.811559 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:17.811532    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28"]
Oct 30 20:15:17.838765 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:17.838725    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z"]
Oct 30 20:15:17.990620 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:17.990229    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8"
Oct 30 20:15:18.090405 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.090381    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28"
Oct 30 20:15:18.094253 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.094171    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z"
Oct 30 20:15:18.103021 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.103001    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f-kube-api-access-62ssf\" (UniqueName: \"kubernetes.io/projected/a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f-kube-api-access-62ssf\") pod \"a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f\" (UID: \"a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f\") "
Oct 30 20:15:18.103165 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.103040    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f-maas-api-service-ca\") pod \"a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f\" (UID: \"a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f\") "
Oct 30 20:15:18.103413 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.103393    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f-maas-api-service-ca" pod "a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f" (UID: "a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:18.103612 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.103596    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:18.105382 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.105358    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f-kube-api-access-62ssf" pod "a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f" (UID: "a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f"). InnerVolumeSpecName "kube-api-access-62ssf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:18.198313 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.195965    6030 scope.go:122] "RemoveContainer" containerID="b675c166f1153715ef35f79f2d86802c9a22c3b173bd1988a9381b26d631c570"
Oct 30 20:15:18.210939 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.210694    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e28b0af-fad5-44bb-b949-08d292e74c97-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e28b0af-fad5-44bb-b949-08d292e74c97-maas-api-service-ca\") pod \"6e28b0af-fad5-44bb-b949-08d292e74c97\" (UID: \"6e28b0af-fad5-44bb-b949-08d292e74c97\") "
Oct 30 20:15:18.210939 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.210742    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca670d4-cee3-4066-80e6-8b3a5ca7783b-kube-api-access-nzlv5\" (UniqueName: \"kubernetes.io/projected/4ca670d4-cee3-4066-80e6-8b3a5ca7783b-kube-api-access-nzlv5\") pod \"4ca670d4-cee3-4066-80e6-8b3a5ca7783b\" (UID: \"4ca670d4-cee3-4066-80e6-8b3a5ca7783b\") "
Oct 30 20:15:18.210939 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.210814    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e28b0af-fad5-44bb-b949-08d292e74c97-kube-api-access-d4mdw\" (UniqueName: \"kubernetes.io/projected/6e28b0af-fad5-44bb-b949-08d292e74c97-kube-api-access-d4mdw\") pod \"6e28b0af-fad5-44bb-b949-08d292e74c97\" (UID: \"6e28b0af-fad5-44bb-b949-08d292e74c97\") "
Oct 30 20:15:18.211286 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.211183    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca670d4-cee3-4066-80e6-8b3a5ca7783b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca670d4-cee3-4066-80e6-8b3a5ca7783b-maas-api-service-ca\") pod \"4ca670d4-cee3-4066-80e6-8b3a5ca7783b\" (UID: \"4ca670d4-cee3-4066-80e6-8b3a5ca7783b\") "
Oct 30 20:15:18.211824 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.211800    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62ssf\" (UniqueName: \"kubernetes.io/projected/a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f-kube-api-access-62ssf\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:18.212698 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.212232    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca670d4-cee3-4066-80e6-8b3a5ca7783b-maas-api-service-ca" pod "4ca670d4-cee3-4066-80e6-8b3a5ca7783b" (UID: "4ca670d4-cee3-4066-80e6-8b3a5ca7783b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:18.212698 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.212493    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e28b0af-fad5-44bb-b949-08d292e74c97-maas-api-service-ca" pod "6e28b0af-fad5-44bb-b949-08d292e74c97" (UID: "6e28b0af-fad5-44bb-b949-08d292e74c97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:18.215999 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.215960    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca670d4-cee3-4066-80e6-8b3a5ca7783b-kube-api-access-nzlv5" pod "4ca670d4-cee3-4066-80e6-8b3a5ca7783b" (UID: "4ca670d4-cee3-4066-80e6-8b3a5ca7783b"). InnerVolumeSpecName "kube-api-access-nzlv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:18.217906 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.217873    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e28b0af-fad5-44bb-b949-08d292e74c97-kube-api-access-d4mdw" pod "6e28b0af-fad5-44bb-b949-08d292e74c97" (UID: "6e28b0af-fad5-44bb-b949-08d292e74c97"). InnerVolumeSpecName "kube-api-access-d4mdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:18.314027 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.313996    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4mdw\" (UniqueName: \"kubernetes.io/projected/6e28b0af-fad5-44bb-b949-08d292e74c97-kube-api-access-d4mdw\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:18.314223 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.314033    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca670d4-cee3-4066-80e6-8b3a5ca7783b-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:18.314223 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.314066    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e28b0af-fad5-44bb-b949-08d292e74c97-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:18.314223 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.314081    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzlv5\" (UniqueName: \"kubernetes.io/projected/4ca670d4-cee3-4066-80e6-8b3a5ca7783b-kube-api-access-nzlv5\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:18.771804 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.771765    6030 generic.go:356] "Generic (PLEG): container finished" podID="17e42313-2fdd-44dc-9f24-07c9f8283813" containerID="ee5b26e56fa9cd89b8b2f5456eef4b7fce699d4377c7900cd37bfa16f91650c0" exitCode=60
Oct 30 20:15:18.772344 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.771869    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t" event={"ID":"17e42313-2fdd-44dc-9f24-07c9f8283813","Type":"ContainerDied","Data":"ee5b26e56fa9cd89b8b2f5456eef4b7fce699d4377c7900cd37bfa16f91650c0"}
Oct 30 20:15:18.772344 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.771985    6030 scope.go:122] "RemoveContainer" containerID="6d3682975c0b387cd8322063c2287c5a708b54282fc7afdd9a53814641c3a379"
Oct 30 20:15:18.773106 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.773075    6030 scope.go:122] "RemoveContainer" containerID="ee5b26e56fa9cd89b8b2f5456eef4b7fce699d4377c7900cd37bfa16f91650c0"
Oct 30 20:15:18.773941 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:18.773515    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t_odh-ai-gateway-infra(17e42313-2fdd-44dc-9f24-07c9f8283813)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t" podUID="17e42313-2fdd-44dc-9f24-07c9f8283813"
Oct 30 20:15:18.774415 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.774399    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28"
Oct 30 20:15:18.774521 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.774419    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28" event={"ID":"4ca670d4-cee3-4066-80e6-8b3a5ca7783b","Type":"ContainerDied","Data":"2ff7c3c2fff669bcad40c9dfefe2d203bcea14de24004984d76f8eb748bf8c22"}
Oct 30 20:15:18.776736 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.776715    6030 generic.go:356] "Generic (PLEG): container finished" podID="75e2878b-a4f8-4180-a5c2-cb169475f2d7" containerID="aa43c36ccbd8d2e638481c57e5cfa634d48a2fb7a87231f343ea340f87e0575b" exitCode=60
Oct 30 20:15:18.776860 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.776788    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7" event={"ID":"75e2878b-a4f8-4180-a5c2-cb169475f2d7","Type":"ContainerDied","Data":"aa43c36ccbd8d2e638481c57e5cfa634d48a2fb7a87231f343ea340f87e0575b"}
Oct 30 20:15:18.777186 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.777168    6030 scope.go:122] "RemoveContainer" containerID="aa43c36ccbd8d2e638481c57e5cfa634d48a2fb7a87231f343ea340f87e0575b"
Oct 30 20:15:18.777447 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:18.777422    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7_odh-ai-gateway-infra(75e2878b-a4f8-4180-a5c2-cb169475f2d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7" podUID="75e2878b-a4f8-4180-a5c2-cb169475f2d7"
Oct 30 20:15:18.781088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.781067    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv" event={"ID":"380956ae-205c-4814-aa2d-68aae67dd2cc","Type":"ContainerStarted","Data":"e3b9c1e9aa307e86b759ab19cd5083fe009480ab0ee4050412e5aafbcde43cf5"}
Oct 30 20:15:18.783673 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.783652    6030 generic.go:356] "Generic (PLEG): container finished" podID="461521b6-3031-46d9-8e54-765c46db9983" containerID="33ff872371ef67fb72a6c07a708a002e1ab89c34464b31cca98cbbfb20a041d7" exitCode=60
Oct 30 20:15:18.783767 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.783742    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2" event={"ID":"461521b6-3031-46d9-8e54-765c46db9983","Type":"ContainerDied","Data":"33ff872371ef67fb72a6c07a708a002e1ab89c34464b31cca98cbbfb20a041d7"}
Oct 30 20:15:18.784036 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.784021    6030 scope.go:122] "RemoveContainer" containerID="33ff872371ef67fb72a6c07a708a002e1ab89c34464b31cca98cbbfb20a041d7"
Oct 30 20:15:18.784273 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:18.784249    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2_odh-ai-gateway-infra(461521b6-3031-46d9-8e54-765c46db9983)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2" podUID="461521b6-3031-46d9-8e54-765c46db9983"
Oct 30 20:15:18.786725 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.786440    6030 generic.go:356] "Generic (PLEG): container finished" podID="9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b" containerID="72c6e78cb6ca5191630930ee416837e93315960d9cbfdb898f8ab09560747caf" exitCode=60
Oct 30 20:15:18.786725 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.786513    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5" event={"ID":"9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b","Type":"ContainerDied","Data":"72c6e78cb6ca5191630930ee416837e93315960d9cbfdb898f8ab09560747caf"}
Oct 30 20:15:18.786889 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.786862    6030 scope.go:122] "RemoveContainer" containerID="72c6e78cb6ca5191630930ee416837e93315960d9cbfdb898f8ab09560747caf"
Oct 30 20:15:18.787198 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:18.787176    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5_odh-ai-gateway-infra(9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5" podUID="9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b"
Oct 30 20:15:18.788141 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.788123    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8"
Oct 30 20:15:18.788236 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.788153    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8" event={"ID":"a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f","Type":"ContainerDied","Data":"aae6336612aa355cfe375e50aaab2db7bc649efefbb6276b0941d57c952c2dcb"}
Oct 30 20:15:18.789691 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.789671    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z" event={"ID":"6e28b0af-fad5-44bb-b949-08d292e74c97","Type":"ContainerDied","Data":"68e9a577bee327c2db9391b46913cb96ca162ee6c4b4b988e4b8178111a26d7d"}
Oct 30 20:15:18.789783 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.789696    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z"
Oct 30 20:15:18.789901 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.789885    6030 scope.go:122] "RemoveContainer" containerID="2b8c552eea88627ac422aa4910ed2c6cdacefa3ad7d504b53a70a9c6b8c21855"
Oct 30 20:15:18.802830 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.802810    6030 scope.go:122] "RemoveContainer" containerID="1c6dc9ec6a2a6aeb403e072f27aa4718ad8bc9fdec96a34463cd758ac804f334"
Oct 30 20:15:18.814979 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.814948    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28"]
Oct 30 20:15:18.819498 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.819474    6030 scope.go:122] "RemoveContainer" containerID="04c833f81579921a113f9b1f99150a65c863e7e7d79d235072dde425fa405f33"
Oct 30 20:15:18.825114 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.824921    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8c8241-29889855-jqb28"]
Oct 30 20:15:18.835179 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.835164    6030 scope.go:122] "RemoveContainer" containerID="1b19e3ebd9bebdd38b060ad6d5ff8008be5c0d3308fb0a7523332901a2eec98a"
Oct 30 20:15:18.850778 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.850758    6030 scope.go:122] "RemoveContainer" containerID="5a162e2f9f237be90ff562b51b73f9e97dbec6c89cebad01577785b68cc089e1"
Oct 30 20:15:18.862615 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.862594    6030 scope.go:122] "RemoveContainer" containerID="d99120e68cbf34e58c8b90416f81d5a2e7eda74bd7bee24ed3a74a7648386a80"
Oct 30 20:15:18.940415 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.940381    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8"]
Oct 30 20:15:18.955827 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.955794    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2a3e1b-29889855-rdvx8"]
Oct 30 20:15:18.973609 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.973582    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z"]
Oct 30 20:15:18.987384 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:18.987351    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7eaeed-29889855-hwp9z"]
Oct 30 20:15:19.210717 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.210681    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca670d4-cee3-4066-80e6-8b3a5ca7783b" path="/var/lib/kubelet/pods/4ca670d4-cee3-4066-80e6-8b3a5ca7783b/volumes"
Oct 30 20:15:19.214241 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.214210    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e28b0af-fad5-44bb-b949-08d292e74c97" path="/var/lib/kubelet/pods/6e28b0af-fad5-44bb-b949-08d292e74c97/volumes"
Oct 30 20:15:19.215074 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.215022    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f" path="/var/lib/kubelet/pods/a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f/volumes"
Oct 30 20:15:19.692238 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.692207    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"]
Oct 30 20:15:19.695440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.695417    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:19.805266 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.805236    6030 generic.go:356] "Generic (PLEG): container finished" podID="380956ae-205c-4814-aa2d-68aae67dd2cc" containerID="e3b9c1e9aa307e86b759ab19cd5083fe009480ab0ee4050412e5aafbcde43cf5" exitCode=60
Oct 30 20:15:19.805438 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.805308    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv" event={"ID":"380956ae-205c-4814-aa2d-68aae67dd2cc","Type":"ContainerDied","Data":"e3b9c1e9aa307e86b759ab19cd5083fe009480ab0ee4050412e5aafbcde43cf5"}
Oct 30 20:15:19.805438 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.805400    6030 scope.go:122] "RemoveContainer" containerID="b675c166f1153715ef35f79f2d86802c9a22c3b173bd1988a9381b26d631c570"
Oct 30 20:15:19.805797 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.805778    6030 scope.go:122] "RemoveContainer" containerID="e3b9c1e9aa307e86b759ab19cd5083fe009480ab0ee4050412e5aafbcde43cf5"
Oct 30 20:15:19.806195 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:19.806159    6030 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-7pmtv_odh-ai-gateway-infra(380956ae-205c-4814-aa2d-68aae67dd2cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv" podUID="380956ae-205c-4814-aa2d-68aae67dd2cc"
Oct 30 20:15:19.812535 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.812515    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t"]
Oct 30 20:15:19.873247 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.873211    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv"]
Oct 30 20:15:19.901200 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.901162    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5"]
Oct 30 20:15:19.918333 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.918303    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"]
Oct 30 20:15:19.920530 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920502    6030 cpu_manager.go:414] "containerMap: removing container" podUID="4ca670d4-cee3-4066-80e6-8b3a5ca7783b" containerName="cleanup"
Oct 30 20:15:19.920530 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920530    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca670d4-cee3-4066-80e6-8b3a5ca7783b" containerName="cleanup"
Oct 30 20:15:19.920692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920562    6030 cpu_manager.go:414] "containerMap: removing container" podUID="4ca670d4-cee3-4066-80e6-8b3a5ca7783b" containerName="cleanup"
Oct 30 20:15:19.920692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920570    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca670d4-cee3-4066-80e6-8b3a5ca7783b" containerName="cleanup"
Oct 30 20:15:19.920692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920586    6030 cpu_manager.go:414] "containerMap: removing container" podUID="4ca670d4-cee3-4066-80e6-8b3a5ca7783b" containerName="cleanup"
Oct 30 20:15:19.920692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920593    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca670d4-cee3-4066-80e6-8b3a5ca7783b" containerName="cleanup"
Oct 30 20:15:19.920692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920604    6030 cpu_manager.go:414] "containerMap: removing container" podUID="a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f" containerName="cleanup"
Oct 30 20:15:19.920692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920611    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f" containerName="cleanup"
Oct 30 20:15:19.920692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920627    6030 cpu_manager.go:414] "containerMap: removing container" podUID="6e28b0af-fad5-44bb-b949-08d292e74c97" containerName="cleanup"
Oct 30 20:15:19.920692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920633    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e28b0af-fad5-44bb-b949-08d292e74c97" containerName="cleanup"
Oct 30 20:15:19.920692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920646    6030 cpu_manager.go:414] "containerMap: removing container" podUID="6e28b0af-fad5-44bb-b949-08d292e74c97" containerName="cleanup"
Oct 30 20:15:19.920692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920654    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e28b0af-fad5-44bb-b949-08d292e74c97" containerName="cleanup"
Oct 30 20:15:19.920692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920669    6030 cpu_manager.go:414] "containerMap: removing container" podUID="a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f" containerName="cleanup"
Oct 30 20:15:19.920692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920676    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f" containerName="cleanup"
Oct 30 20:15:19.921088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920716    6030 cpu_manager.go:414] "containerMap: removing container" podUID="a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f" containerName="cleanup"
Oct 30 20:15:19.921088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920723    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f" containerName="cleanup"
Oct 30 20:15:19.921088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920904    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e28b0af-fad5-44bb-b949-08d292e74c97" containerName="cleanup"
Oct 30 20:15:19.921088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920916    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f" containerName="cleanup"
Oct 30 20:15:19.921088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920934    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca670d4-cee3-4066-80e6-8b3a5ca7783b" containerName="cleanup"
Oct 30 20:15:19.921088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920946    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca670d4-cee3-4066-80e6-8b3a5ca7783b" containerName="cleanup"
Oct 30 20:15:19.921088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920958    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca670d4-cee3-4066-80e6-8b3a5ca7783b" containerName="cleanup"
Oct 30 20:15:19.921088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920969    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e28b0af-fad5-44bb-b949-08d292e74c97" containerName="cleanup"
Oct 30 20:15:19.921088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920978    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e28b0af-fad5-44bb-b949-08d292e74c97" containerName="cleanup"
Oct 30 20:15:19.921088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:19.920987    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f" containerName="cleanup"
Oct 30 20:15:20.014002 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.013925    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2"]
Oct 30 20:15:20.014282 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.014263    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"]
Oct 30 20:15:20.014459 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.014445    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7"]
Oct 30 20:15:20.021225 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.014850    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"
Oct 30 20:15:20.084671 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.062249    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:15:20.204080 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.194912    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t"
Oct 30 20:15:20.223073 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.221656    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a441f37d-5657-4821-bf5e-8a9c62e9eb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfd0074b-gshn6\" (UID: \"a441f37d-5657-4821-bf5e-8a9c62e9eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"
Oct 30 20:15:20.223073 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.221712    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a441f37d-5657-4821-bf5e-8a9c62e9eb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfd0074b-gshn6\" (UID: \"a441f37d-5657-4821-bf5e-8a9c62e9eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"
Oct 30 20:15:20.223073 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.221749    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwlg6\" (UniqueName: \"kubernetes.io/projected/a441f37d-5657-4821-bf5e-8a9c62e9eb86-kube-api-access-mwlg6\") pod \"maas-api-revoke-sub-cfd0074b-gshn6\" (UID: \"a441f37d-5657-4821-bf5e-8a9c62e9eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"
Oct 30 20:15:20.328185 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.327509    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e42313-2fdd-44dc-9f24-07c9f8283813-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e42313-2fdd-44dc-9f24-07c9f8283813-maas-api-service-ca\") pod \"17e42313-2fdd-44dc-9f24-07c9f8283813\" (UID: \"17e42313-2fdd-44dc-9f24-07c9f8283813\") "
Oct 30 20:15:20.328185 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.327770    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e42313-2fdd-44dc-9f24-07c9f8283813-kube-api-access-tbf4r\" (UniqueName: \"kubernetes.io/projected/17e42313-2fdd-44dc-9f24-07c9f8283813-kube-api-access-tbf4r\") pod \"17e42313-2fdd-44dc-9f24-07c9f8283813\" (UID: \"17e42313-2fdd-44dc-9f24-07c9f8283813\") "
Oct 30 20:15:20.328397 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.328227    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a441f37d-5657-4821-bf5e-8a9c62e9eb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfd0074b-gshn6\" (UID: \"a441f37d-5657-4821-bf5e-8a9c62e9eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"
Oct 30 20:15:20.328397 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.328269    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a441f37d-5657-4821-bf5e-8a9c62e9eb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfd0074b-gshn6\" (UID: \"a441f37d-5657-4821-bf5e-8a9c62e9eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"
Oct 30 20:15:20.328397 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.328300    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwlg6\" (UniqueName: \"kubernetes.io/projected/a441f37d-5657-4821-bf5e-8a9c62e9eb86-kube-api-access-mwlg6\") pod \"maas-api-revoke-sub-cfd0074b-gshn6\" (UID: \"a441f37d-5657-4821-bf5e-8a9c62e9eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"
Oct 30 20:15:20.334633 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.332413    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a441f37d-5657-4821-bf5e-8a9c62e9eb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfd0074b-gshn6\" (UID: \"a441f37d-5657-4821-bf5e-8a9c62e9eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"
Oct 30 20:15:20.339030 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.337774    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a441f37d-5657-4821-bf5e-8a9c62e9eb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfd0074b-gshn6\" (UID: \"a441f37d-5657-4821-bf5e-8a9c62e9eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"
Oct 30 20:15:20.344326 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.344283    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e42313-2fdd-44dc-9f24-07c9f8283813-maas-api-service-ca" pod "17e42313-2fdd-44dc-9f24-07c9f8283813" (UID: "17e42313-2fdd-44dc-9f24-07c9f8283813"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:20.347538 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.347279    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e42313-2fdd-44dc-9f24-07c9f8283813-kube-api-access-tbf4r" pod "17e42313-2fdd-44dc-9f24-07c9f8283813" (UID: "17e42313-2fdd-44dc-9f24-07c9f8283813"). InnerVolumeSpecName "kube-api-access-tbf4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:20.360243 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.356885    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwlg6\" (UniqueName: \"kubernetes.io/projected/a441f37d-5657-4821-bf5e-8a9c62e9eb86-kube-api-access-mwlg6\") pod \"maas-api-revoke-sub-cfd0074b-gshn6\" (UID: \"a441f37d-5657-4821-bf5e-8a9c62e9eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"
Oct 30 20:15:20.418660 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.418625    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5"
Oct 30 20:15:20.438564 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.438523    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e42313-2fdd-44dc-9f24-07c9f8283813-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:20.438726 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.438569    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbf4r\" (UniqueName: \"kubernetes.io/projected/17e42313-2fdd-44dc-9f24-07c9f8283813-kube-api-access-tbf4r\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:20.462700 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.462644    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"
Oct 30 20:15:20.483407 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.483375    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2"
Oct 30 20:15:20.537038 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.537008    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7"
Oct 30 20:15:20.539564 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.539529    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/461521b6-3031-46d9-8e54-765c46db9983-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/461521b6-3031-46d9-8e54-765c46db9983-maas-api-service-ca\") pod \"461521b6-3031-46d9-8e54-765c46db9983\" (UID: \"461521b6-3031-46d9-8e54-765c46db9983\") "
Oct 30 20:15:20.539725 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.539620    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b-kube-api-access-6xc6k\" (UniqueName: \"kubernetes.io/projected/9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b-kube-api-access-6xc6k\") pod \"9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b\" (UID: \"9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b\") "
Oct 30 20:15:20.539725 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.539684    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e2878b-a4f8-4180-a5c2-cb169475f2d7-kube-api-access-b4z97\" (UniqueName: \"kubernetes.io/projected/75e2878b-a4f8-4180-a5c2-cb169475f2d7-kube-api-access-b4z97\") pod \"75e2878b-a4f8-4180-a5c2-cb169475f2d7\" (UID: \"75e2878b-a4f8-4180-a5c2-cb169475f2d7\") "
Oct 30 20:15:20.539844 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.539754    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b-maas-api-service-ca\") pod \"9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b\" (UID: \"9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b\") "
Oct 30 20:15:20.539844 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.539787    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e2878b-a4f8-4180-a5c2-cb169475f2d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e2878b-a4f8-4180-a5c2-cb169475f2d7-maas-api-service-ca\") pod \"75e2878b-a4f8-4180-a5c2-cb169475f2d7\" (UID: \"75e2878b-a4f8-4180-a5c2-cb169475f2d7\") "
Oct 30 20:15:20.540030 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.540006    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/461521b6-3031-46d9-8e54-765c46db9983-kube-api-access-4szcq\" (UniqueName: \"kubernetes.io/projected/461521b6-3031-46d9-8e54-765c46db9983-kube-api-access-4szcq\") pod \"461521b6-3031-46d9-8e54-765c46db9983\" (UID: \"461521b6-3031-46d9-8e54-765c46db9983\") "
Oct 30 20:15:20.540200 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.540178    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/461521b6-3031-46d9-8e54-765c46db9983-maas-api-service-ca" pod "461521b6-3031-46d9-8e54-765c46db9983" (UID: "461521b6-3031-46d9-8e54-765c46db9983"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:20.540560 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.540533    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b-maas-api-service-ca" pod "9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b" (UID: "9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:20.540696 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.540655    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/461521b6-3031-46d9-8e54-765c46db9983-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:20.540696 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.540542    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e2878b-a4f8-4180-a5c2-cb169475f2d7-maas-api-service-ca" pod "75e2878b-a4f8-4180-a5c2-cb169475f2d7" (UID: "75e2878b-a4f8-4180-a5c2-cb169475f2d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:20.545557 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.545478    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b-kube-api-access-6xc6k" pod "9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b" (UID: "9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b"). InnerVolumeSpecName "kube-api-access-6xc6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:20.545861 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.545833    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461521b6-3031-46d9-8e54-765c46db9983-kube-api-access-4szcq" pod "461521b6-3031-46d9-8e54-765c46db9983" (UID: "461521b6-3031-46d9-8e54-765c46db9983"). InnerVolumeSpecName "kube-api-access-4szcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:20.549331 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.549274    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e2878b-a4f8-4180-a5c2-cb169475f2d7-kube-api-access-b4z97" pod "75e2878b-a4f8-4180-a5c2-cb169475f2d7" (UID: "75e2878b-a4f8-4180-a5c2-cb169475f2d7"). InnerVolumeSpecName "kube-api-access-b4z97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:20.642399 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.642361    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4szcq\" (UniqueName: \"kubernetes.io/projected/461521b6-3031-46d9-8e54-765c46db9983-kube-api-access-4szcq\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:20.642399 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.642401    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xc6k\" (UniqueName: \"kubernetes.io/projected/9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b-kube-api-access-6xc6k\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:20.642568 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.642417    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4z97\" (UniqueName: \"kubernetes.io/projected/75e2878b-a4f8-4180-a5c2-cb169475f2d7-kube-api-access-b4z97\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:20.642568 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.642432    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:20.642568 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.642446    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e2878b-a4f8-4180-a5c2-cb169475f2d7-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:20.811788 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.811748    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"]
Oct 30 20:15:20.812240 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:15:20.812212    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda441f37d_5657_4821_bf5e_8a9c62e9eb86.slice/crio-abbcf143c74f49c6c7af12b367a11b53b3f4f4f0605f21709e38abb734ac83fb WatchSource:0}: Error finding container abbcf143c74f49c6c7af12b367a11b53b3f4f4f0605f21709e38abb734ac83fb: Status 404 returned error can't find the container with id abbcf143c74f49c6c7af12b367a11b53b3f4f4f0605f21709e38abb734ac83fb
Oct 30 20:15:20.819234 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.819203    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t" event={"ID":"17e42313-2fdd-44dc-9f24-07c9f8283813","Type":"ContainerDied","Data":"396f1df24c15ef6994b8045267199b974bb75d183e50afa1d5252c07b6649de8"}
Oct 30 20:15:20.819343 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.819230    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t"
Oct 30 20:15:20.819343 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.819270    6030 scope.go:122] "RemoveContainer" containerID="ee5b26e56fa9cd89b8b2f5456eef4b7fce699d4377c7900cd37bfa16f91650c0"
Oct 30 20:15:20.823261 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.821092    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7" event={"ID":"75e2878b-a4f8-4180-a5c2-cb169475f2d7","Type":"ContainerDied","Data":"6d30d4ba06b12d39d232dd6d3701edd8555bbc1068928739005d0259c9587014"}
Oct 30 20:15:20.823261 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.821110    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7"
Oct 30 20:15:20.826926 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.825533    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2" event={"ID":"461521b6-3031-46d9-8e54-765c46db9983","Type":"ContainerDied","Data":"03d64cc8f1928c8efa0127fe41d1b457efabf55aa4f49750f4ed9d3e654a723f"}
Oct 30 20:15:20.826926 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.825608    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2"
Oct 30 20:15:20.832971 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.832945    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5" event={"ID":"9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b","Type":"ContainerDied","Data":"d621390c6842abffda1d3253c3afad906a49005a95f4bac379a0a3cf6564cb47"}
Oct 30 20:15:20.833216 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.833036    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5"
Oct 30 20:15:20.835604 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.835319    6030 scope.go:122] "RemoveContainer" containerID="aa43c36ccbd8d2e638481c57e5cfa634d48a2fb7a87231f343ea340f87e0575b"
Oct 30 20:15:20.855473 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.855450    6030 scope.go:122] "RemoveContainer" containerID="33ff872371ef67fb72a6c07a708a002e1ab89c34464b31cca98cbbfb20a041d7"
Oct 30 20:15:20.868681 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.868658    6030 scope.go:122] "RemoveContainer" containerID="72c6e78cb6ca5191630930ee416837e93315960d9cbfdb898f8ab09560747caf"
Oct 30 20:15:20.882167 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.882138    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5"]
Oct 30 20:15:20.885816 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.885793    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889855-xgqc5"]
Oct 30 20:15:20.912077 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.912025    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2"]
Oct 30 20:15:20.924934 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.924903    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b24a1a-29889855-sqsp2"]
Oct 30 20:15:20.944127 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.944085    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7"]
Oct 30 20:15:20.952799 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.952764    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889855-2lbj7"]
Oct 30 20:15:20.980403 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.980369    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t"]
Oct 30 20:15:20.986713 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:20.986683    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f06af6-29889855-jtw6t"]
Oct 30 20:15:21.081013 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.080990    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv"
Oct 30 20:15:21.152601 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.152571    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/380956ae-205c-4814-aa2d-68aae67dd2cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380956ae-205c-4814-aa2d-68aae67dd2cc-maas-api-service-ca\") pod \"380956ae-205c-4814-aa2d-68aae67dd2cc\" (UID: \"380956ae-205c-4814-aa2d-68aae67dd2cc\") "
Oct 30 20:15:21.152848 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.152827    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/380956ae-205c-4814-aa2d-68aae67dd2cc-kube-api-access-59l2f\" (UniqueName: \"kubernetes.io/projected/380956ae-205c-4814-aa2d-68aae67dd2cc-kube-api-access-59l2f\") pod \"380956ae-205c-4814-aa2d-68aae67dd2cc\" (UID: \"380956ae-205c-4814-aa2d-68aae67dd2cc\") "
Oct 30 20:15:21.152955 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.152937    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380956ae-205c-4814-aa2d-68aae67dd2cc-maas-api-service-ca" pod "380956ae-205c-4814-aa2d-68aae67dd2cc" (UID: "380956ae-205c-4814-aa2d-68aae67dd2cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:21.153337 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.153313    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380956ae-205c-4814-aa2d-68aae67dd2cc-maas-api-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:21.170100 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.170039    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380956ae-205c-4814-aa2d-68aae67dd2cc-kube-api-access-59l2f" pod "380956ae-205c-4814-aa2d-68aae67dd2cc" (UID: "380956ae-205c-4814-aa2d-68aae67dd2cc"). InnerVolumeSpecName "kube-api-access-59l2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:21.204139 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.204110    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e42313-2fdd-44dc-9f24-07c9f8283813" path="/var/lib/kubelet/pods/17e42313-2fdd-44dc-9f24-07c9f8283813/volumes"
Oct 30 20:15:21.204580 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.204566    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="461521b6-3031-46d9-8e54-765c46db9983" path="/var/lib/kubelet/pods/461521b6-3031-46d9-8e54-765c46db9983/volumes"
Oct 30 20:15:21.204982 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.204968    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e2878b-a4f8-4180-a5c2-cb169475f2d7" path="/var/lib/kubelet/pods/75e2878b-a4f8-4180-a5c2-cb169475f2d7/volumes"
Oct 30 20:15:21.205693 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.205674    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b" path="/var/lib/kubelet/pods/9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b/volumes"
Oct 30 20:15:21.254584 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.254552    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59l2f\" (UniqueName: \"kubernetes.io/projected/380956ae-205c-4814-aa2d-68aae67dd2cc-kube-api-access-59l2f\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:21.843320 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.843288    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv"
Oct 30 20:15:21.843320 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.843290    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv" event={"ID":"380956ae-205c-4814-aa2d-68aae67dd2cc","Type":"ContainerDied","Data":"0ce340e7d145a8a7abec91cd7f5bbc04e2cabd69e023df4e64f1ceb856635902"}
Oct 30 20:15:21.843551 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.843357    6030 scope.go:122] "RemoveContainer" containerID="e3b9c1e9aa307e86b759ab19cd5083fe009480ab0ee4050412e5aafbcde43cf5"
Oct 30 20:15:21.845244 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.845216    6030 generic.go:356] "Generic (PLEG): container finished" podID="a441f37d-5657-4821-bf5e-8a9c62e9eb86" containerID="ce3c52828a3f5dcbcfa5122852852de805238921c2e0dc1399526d3dfe53e751" exitCode=0
Oct 30 20:15:21.845384 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.845303    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6" event={"ID":"a441f37d-5657-4821-bf5e-8a9c62e9eb86","Type":"ContainerDied","Data":"ce3c52828a3f5dcbcfa5122852852de805238921c2e0dc1399526d3dfe53e751"}
Oct 30 20:15:21.845384 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.845355    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6" event={"ID":"a441f37d-5657-4821-bf5e-8a9c62e9eb86","Type":"ContainerStarted","Data":"abbcf143c74f49c6c7af12b367a11b53b3f4f4f0605f21709e38abb734ac83fb"}
Oct 30 20:15:21.900617 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.900575    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv"]
Oct 30 20:15:21.908706 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:21.908664    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-7pmtv"]
Oct 30 20:15:23.058078 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:23.058035    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"
Oct 30 20:15:23.084608 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:23.084579    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a441f37d-5657-4821-bf5e-8a9c62e9eb86-kube-api-access-mwlg6\" (UniqueName: \"kubernetes.io/projected/a441f37d-5657-4821-bf5e-8a9c62e9eb86-kube-api-access-mwlg6\") pod \"a441f37d-5657-4821-bf5e-8a9c62e9eb86\" (UID: \"a441f37d-5657-4821-bf5e-8a9c62e9eb86\") "
Oct 30 20:15:23.084829 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:23.084653    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a441f37d-5657-4821-bf5e-8a9c62e9eb86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a441f37d-5657-4821-bf5e-8a9c62e9eb86-maas-api-openshift-ca\") pod \"a441f37d-5657-4821-bf5e-8a9c62e9eb86\" (UID: \"a441f37d-5657-4821-bf5e-8a9c62e9eb86\") "
Oct 30 20:15:23.084901 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:23.084833    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a441f37d-5657-4821-bf5e-8a9c62e9eb86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a441f37d-5657-4821-bf5e-8a9c62e9eb86-maas-api-xks-ca\") pod \"a441f37d-5657-4821-bf5e-8a9c62e9eb86\" (UID: \"a441f37d-5657-4821-bf5e-8a9c62e9eb86\") "
Oct 30 20:15:23.085017 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:23.084995    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a441f37d-5657-4821-bf5e-8a9c62e9eb86-maas-api-openshift-ca" pod "a441f37d-5657-4821-bf5e-8a9c62e9eb86" (UID: "a441f37d-5657-4821-bf5e-8a9c62e9eb86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:23.085416 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:23.085397    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a441f37d-5657-4821-bf5e-8a9c62e9eb86-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:23.086847 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:23.086817    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a441f37d-5657-4821-bf5e-8a9c62e9eb86-kube-api-access-mwlg6" pod "a441f37d-5657-4821-bf5e-8a9c62e9eb86" (UID: "a441f37d-5657-4821-bf5e-8a9c62e9eb86"). InnerVolumeSpecName "kube-api-access-mwlg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:23.087087 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:23.087066    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a441f37d-5657-4821-bf5e-8a9c62e9eb86-maas-api-xks-ca" pod "a441f37d-5657-4821-bf5e-8a9c62e9eb86" (UID: "a441f37d-5657-4821-bf5e-8a9c62e9eb86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:23.187071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:23.187010    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a441f37d-5657-4821-bf5e-8a9c62e9eb86-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:23.187071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:23.187038    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwlg6\" (UniqueName: \"kubernetes.io/projected/a441f37d-5657-4821-bf5e-8a9c62e9eb86-kube-api-access-mwlg6\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:23.204433 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:23.204406    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="380956ae-205c-4814-aa2d-68aae67dd2cc" path="/var/lib/kubelet/pods/380956ae-205c-4814-aa2d-68aae67dd2cc/volumes"
Oct 30 20:15:23.867843 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:23.867810    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"
Oct 30 20:15:23.868099 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:23.867836    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6" event={"ID":"a441f37d-5657-4821-bf5e-8a9c62e9eb86","Type":"ContainerDied","Data":"abbcf143c74f49c6c7af12b367a11b53b3f4f4f0605f21709e38abb734ac83fb"}
Oct 30 20:15:23.868099 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:23.867893    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abbcf143c74f49c6c7af12b367a11b53b3f4f4f0605f21709e38abb734ac83fb"
Oct 30 20:15:24.692581 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:24.692536    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z" podUID="674f52f4-ad53-45f5-90ae-93ac0848ee23" containerName="payload-processing" containerID="cri-o://4ff0eccd6e9663c1ad996a8807a3119f957cb60ab65a4a8c4cdb3d2ae1290b7c" gracePeriod=25
Oct 30 20:15:24.887430 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:24.887397    6030 generic.go:356] "Generic (PLEG): container finished" podID="674f52f4-ad53-45f5-90ae-93ac0848ee23" containerID="4ff0eccd6e9663c1ad996a8807a3119f957cb60ab65a4a8c4cdb3d2ae1290b7c" exitCode=0
Oct 30 20:15:24.887638 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:24.887612    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z" event={"ID":"674f52f4-ad53-45f5-90ae-93ac0848ee23","Type":"ContainerDied","Data":"4ff0eccd6e9663c1ad996a8807a3119f957cb60ab65a4a8c4cdb3d2ae1290b7c"}
Oct 30 20:15:24.970208 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:24.970175    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:25.012401 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:25.012363    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/674f52f4-ad53-45f5-90ae-93ac0848ee23-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/674f52f4-ad53-45f5-90ae-93ac0848ee23-routing-overlay-volume\") pod \"674f52f4-ad53-45f5-90ae-93ac0848ee23\" (UID: \"674f52f4-ad53-45f5-90ae-93ac0848ee23\") "
Oct 30 20:15:25.012616 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:25.012601    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/674f52f4-ad53-45f5-90ae-93ac0848ee23-provider-credentials\" (UniqueName: \"kubernetes.io/projected/674f52f4-ad53-45f5-90ae-93ac0848ee23-provider-credentials\") pod \"674f52f4-ad53-45f5-90ae-93ac0848ee23\" (UID: \"674f52f4-ad53-45f5-90ae-93ac0848ee23\") "
Oct 30 20:15:25.012698 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:25.012682    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/674f52f4-ad53-45f5-90ae-93ac0848ee23-kube-api-access-wl8qh\" (UniqueName: \"kubernetes.io/projected/674f52f4-ad53-45f5-90ae-93ac0848ee23-kube-api-access-wl8qh\") pod \"674f52f4-ad53-45f5-90ae-93ac0848ee23\" (UID: \"674f52f4-ad53-45f5-90ae-93ac0848ee23\") "
Oct 30 20:15:25.012764 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:25.012716    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/674f52f4-ad53-45f5-90ae-93ac0848ee23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/674f52f4-ad53-45f5-90ae-93ac0848ee23-plugins-config-volume\") pod \"674f52f4-ad53-45f5-90ae-93ac0848ee23\" (UID: \"674f52f4-ad53-45f5-90ae-93ac0848ee23\") "
Oct 30 20:15:25.013265 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:25.013234    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/674f52f4-ad53-45f5-90ae-93ac0848ee23-plugins-config-volume" pod "674f52f4-ad53-45f5-90ae-93ac0848ee23" (UID: "674f52f4-ad53-45f5-90ae-93ac0848ee23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:25.013411 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:25.013391    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/674f52f4-ad53-45f5-90ae-93ac0848ee23-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:25.013482 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:25.013469    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/674f52f4-ad53-45f5-90ae-93ac0848ee23-routing-overlay-volume" pod "674f52f4-ad53-45f5-90ae-93ac0848ee23" (UID: "674f52f4-ad53-45f5-90ae-93ac0848ee23"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:25.015138 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:25.015109    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674f52f4-ad53-45f5-90ae-93ac0848ee23-provider-credentials" pod "674f52f4-ad53-45f5-90ae-93ac0848ee23" (UID: "674f52f4-ad53-45f5-90ae-93ac0848ee23"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:25.015470 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:25.015444    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674f52f4-ad53-45f5-90ae-93ac0848ee23-kube-api-access-wl8qh" pod "674f52f4-ad53-45f5-90ae-93ac0848ee23" (UID: "674f52f4-ad53-45f5-90ae-93ac0848ee23"). InnerVolumeSpecName "kube-api-access-wl8qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:25.114938 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:25.114907    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/674f52f4-ad53-45f5-90ae-93ac0848ee23-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:25.114938 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:25.114935    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl8qh\" (UniqueName: \"kubernetes.io/projected/674f52f4-ad53-45f5-90ae-93ac0848ee23-kube-api-access-wl8qh\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:25.114938 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:25.114947    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/674f52f4-ad53-45f5-90ae-93ac0848ee23-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:25.896411 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:25.896379    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"
Oct 30 20:15:25.896898 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:25.896378    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z" event={"ID":"674f52f4-ad53-45f5-90ae-93ac0848ee23","Type":"ContainerDied","Data":"5780c0eecadc318118c79335effa8c5263b47fec078abb555f3161ebc3642712"}
Oct 30 20:15:25.896898 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:25.896540    6030 scope.go:122] "RemoveContainer" containerID="4ff0eccd6e9663c1ad996a8807a3119f957cb60ab65a4a8c4cdb3d2ae1290b7c"
Oct 30 20:15:25.921371 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:25.921346    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"]
Oct 30 20:15:25.933036 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:25.933012    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-65bck6h2z"]
Oct 30 20:15:27.202924 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:27.202889    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="674f52f4-ad53-45f5-90ae-93ac0848ee23" path="/var/lib/kubelet/pods/674f52f4-ad53-45f5-90ae-93ac0848ee23/volumes"
Oct 30 20:15:34.289007 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:34.288969    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:15:41.118057 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:41.117958    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"]
Oct 30 20:15:42.690237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.690198    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"]
Oct 30 20:15:42.691936 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.691903    6030 cpu_manager.go:414] "containerMap: removing container" podUID="17e42313-2fdd-44dc-9f24-07c9f8283813" containerName="cleanup"
Oct 30 20:15:42.703040 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703006    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e42313-2fdd-44dc-9f24-07c9f8283813" containerName="cleanup"
Oct 30 20:15:42.703230 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703073    6030 cpu_manager.go:414] "containerMap: removing container" podUID="75e2878b-a4f8-4180-a5c2-cb169475f2d7" containerName="cleanup"
Oct 30 20:15:42.703230 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703084    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e2878b-a4f8-4180-a5c2-cb169475f2d7" containerName="cleanup"
Oct 30 20:15:42.703230 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703128    6030 cpu_manager.go:414] "containerMap: removing container" podUID="380956ae-205c-4814-aa2d-68aae67dd2cc" containerName="cleanup"
Oct 30 20:15:42.703230 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703138    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380956ae-205c-4814-aa2d-68aae67dd2cc" containerName="cleanup"
Oct 30 20:15:42.703230 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703152    6030 cpu_manager.go:414] "containerMap: removing container" podUID="75e2878b-a4f8-4180-a5c2-cb169475f2d7" containerName="cleanup"
Oct 30 20:15:42.703230 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703159    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e2878b-a4f8-4180-a5c2-cb169475f2d7" containerName="cleanup"
Oct 30 20:15:42.703230 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703175    6030 cpu_manager.go:414] "containerMap: removing container" podUID="461521b6-3031-46d9-8e54-765c46db9983" containerName="cleanup"
Oct 30 20:15:42.703230 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703182    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461521b6-3031-46d9-8e54-765c46db9983" containerName="cleanup"
Oct 30 20:15:42.703230 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703193    6030 cpu_manager.go:414] "containerMap: removing container" podUID="674f52f4-ad53-45f5-90ae-93ac0848ee23" containerName="payload-processing"
Oct 30 20:15:42.703230 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703201    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674f52f4-ad53-45f5-90ae-93ac0848ee23" containerName="payload-processing"
Oct 30 20:15:42.703230 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703223    6030 cpu_manager.go:414] "containerMap: removing container" podUID="75e2878b-a4f8-4180-a5c2-cb169475f2d7" containerName="cleanup"
Oct 30 20:15:42.703230 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703231    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e2878b-a4f8-4180-a5c2-cb169475f2d7" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703260    6030 cpu_manager.go:414] "containerMap: removing container" podUID="6e28b0af-fad5-44bb-b949-08d292e74c97" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703266    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e28b0af-fad5-44bb-b949-08d292e74c97" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703281    6030 cpu_manager.go:414] "containerMap: removing container" podUID="17e42313-2fdd-44dc-9f24-07c9f8283813" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703287    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e42313-2fdd-44dc-9f24-07c9f8283813" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703298    6030 cpu_manager.go:414] "containerMap: removing container" podUID="461521b6-3031-46d9-8e54-765c46db9983" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703304    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461521b6-3031-46d9-8e54-765c46db9983" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703317    6030 cpu_manager.go:414] "containerMap: removing container" podUID="9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703325    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703341    6030 cpu_manager.go:414] "containerMap: removing container" podUID="461521b6-3031-46d9-8e54-765c46db9983" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703349    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461521b6-3031-46d9-8e54-765c46db9983" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703364    6030 cpu_manager.go:414] "containerMap: removing container" podUID="380956ae-205c-4814-aa2d-68aae67dd2cc" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703371    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380956ae-205c-4814-aa2d-68aae67dd2cc" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703385    6030 cpu_manager.go:414] "containerMap: removing container" podUID="a441f37d-5657-4821-bf5e-8a9c62e9eb86" containerName="revoke-keys"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703391    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a441f37d-5657-4821-bf5e-8a9c62e9eb86" containerName="revoke-keys"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703403    6030 cpu_manager.go:414] "containerMap: removing container" podUID="380956ae-205c-4814-aa2d-68aae67dd2cc" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703411    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380956ae-205c-4814-aa2d-68aae67dd2cc" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703424    6030 cpu_manager.go:414] "containerMap: removing container" podUID="9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703431    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703449    6030 cpu_manager.go:414] "containerMap: removing container" podUID="17e42313-2fdd-44dc-9f24-07c9f8283813" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703456    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e42313-2fdd-44dc-9f24-07c9f8283813" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703688    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e42313-2fdd-44dc-9f24-07c9f8283813" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703702    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="461521b6-3031-46d9-8e54-765c46db9983" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703713    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703725    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="461521b6-3031-46d9-8e54-765c46db9983" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703735    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="380956ae-205c-4814-aa2d-68aae67dd2cc" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703747    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="a441f37d-5657-4821-bf5e-8a9c62e9eb86" containerName="revoke-keys"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703757    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703766    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="380956ae-205c-4814-aa2d-68aae67dd2cc" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703774    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703785    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a3b6f0-3fb7-4eb3-aba4-f2966f80bb7f" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703794    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e42313-2fdd-44dc-9f24-07c9f8283813" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703808    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e42313-2fdd-44dc-9f24-07c9f8283813" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703817    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e2878b-a4f8-4180-a5c2-cb169475f2d7" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703831    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e2878b-a4f8-4180-a5c2-cb169475f2d7" containerName="cleanup"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703840    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="674f52f4-ad53-45f5-90ae-93ac0848ee23" containerName="payload-processing"
Oct 30 20:15:42.703863 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.703851    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e2878b-a4f8-4180-a5c2-cb169475f2d7" containerName="cleanup"
Oct 30 20:15:42.710016 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.709991    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"
Oct 30 20:15:42.714260 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.714231    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"]
Oct 30 20:15:42.759969 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.759938    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z47qw\" (UniqueName: \"kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-kube-api-access-z47qw\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr\" (UID: \"79631c91-1914-4a7b-81ac-5b821aefbb6d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"
Oct 30 20:15:42.760163 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.760024    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr\" (UID: \"79631c91-1914-4a7b-81ac-5b821aefbb6d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"
Oct 30 20:15:42.760163 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.760152    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr\" (UID: \"79631c91-1914-4a7b-81ac-5b821aefbb6d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"
Oct 30 20:15:42.760315 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.760200    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr\" (UID: \"79631c91-1914-4a7b-81ac-5b821aefbb6d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"
Oct 30 20:15:42.861984 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.861946    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z47qw\" (UniqueName: \"kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-kube-api-access-z47qw\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr\" (UID: \"79631c91-1914-4a7b-81ac-5b821aefbb6d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"
Oct 30 20:15:42.862275 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.862063    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr\" (UID: \"79631c91-1914-4a7b-81ac-5b821aefbb6d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"
Oct 30 20:15:42.862275 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.862172    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr\" (UID: \"79631c91-1914-4a7b-81ac-5b821aefbb6d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"
Oct 30 20:15:42.862275 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.862230    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr\" (UID: \"79631c91-1914-4a7b-81ac-5b821aefbb6d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"
Oct 30 20:15:42.863241 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.863210    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr\" (UID: \"79631c91-1914-4a7b-81ac-5b821aefbb6d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"
Oct 30 20:15:42.863389 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.863278    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr\" (UID: \"79631c91-1914-4a7b-81ac-5b821aefbb6d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"
Oct 30 20:15:42.866353 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.866325    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr\" (UID: \"79631c91-1914-4a7b-81ac-5b821aefbb6d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"
Oct 30 20:15:42.881303 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:42.881274    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z47qw\" (UniqueName: \"kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-kube-api-access-z47qw\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr\" (UID: \"79631c91-1914-4a7b-81ac-5b821aefbb6d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"
Oct 30 20:15:43.034124 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:43.034026    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"
Oct 30 20:15:43.353140 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:43.352749    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"]
Oct 30 20:15:44.055852 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:44.055816    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr" event={"ID":"79631c91-1914-4a7b-81ac-5b821aefbb6d","Type":"ContainerStarted","Data":"c7a606b9d31ca3454324a821532ff664d1310b30293f65a220899cd0026ae529"}
Oct 30 20:15:44.056391 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:44.055885    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr" event={"ID":"79631c91-1914-4a7b-81ac-5b821aefbb6d","Type":"ContainerStarted","Data":"2a9e60fcdb353e24584566bb4e124b1095058cbb4cd98acf54201c352e20c612"}
Oct 30 20:15:44.056391 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:44.055916    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"
Oct 30 20:15:44.107064 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:44.104542    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr" podStartSLOduration=2.104522484 podStartE2EDuration="2.104522484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:44.101904278 +0000 UTC m=+2623.017511158" watchObservedRunningTime="2026-10-30 20:15:44.104522484 +0000 UTC m=+2623.020129342"
Oct 30 20:15:44.209119 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:44.209069    6030 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d97cc8/agc-cred-52dc941f181c: secret "agc-cred-52dc941f181c" not found
Oct 30 20:15:44.209119 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:44.209101    6030 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr: secret "agc-cred-52dc941f181c" not found
Oct 30 20:15:44.209378 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:44.209169    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-provider-credentials podName:79631c91-1914-4a7b-81ac-5b821aefbb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:15:44.709148646 +0000 UTC m=+2623.624755501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr" (UID: "79631c91-1914-4a7b-81ac-5b821aefbb6d") : secret "agc-cred-52dc941f181c" not found
Oct 30 20:15:44.597359 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:44.597321    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"]
Oct 30 20:15:44.719234 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:44.719204    6030 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d97cc8/agc-cred-52dc941f181c: secret "agc-cred-52dc941f181c" not found
Oct 30 20:15:44.719234 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:44.719232    6030 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr: secret "agc-cred-52dc941f181c" not found
Oct 30 20:15:44.719449 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:44.719303    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-provider-credentials podName:79631c91-1914-4a7b-81ac-5b821aefbb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:15:45.719282644 +0000 UTC m=+2624.634889492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr" (UID: "79631c91-1914-4a7b-81ac-5b821aefbb6d") : secret "agc-cred-52dc941f181c" not found
Oct 30 20:15:45.102958 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:45.102923    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"]
Oct 30 20:15:45.108733 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:45.108700    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"]
Oct 30 20:15:45.227191 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:45.227161    6030 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-plugins-e2e-extmod-b-d97cc8: configmap "payload-processing-external-model-plugins-e2e-extmod-b-d97cc8" not found
Oct 30 20:15:45.227404 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:45.227230    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-plugins-config-volume podName:79631c91-1914-4a7b-81ac-5b821aefbb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:15:45.7272165 +0000 UTC m=+2624.642823367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr" (UID: "79631c91-1914-4a7b-81ac-5b821aefbb6d") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-d97cc8" not found
Oct 30 20:15:45.748669 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:45.748633    6030 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-plugins-e2e-extmod-b-d97cc8: configmap "payload-processing-external-model-plugins-e2e-extmod-b-d97cc8" not found
Oct 30 20:15:45.748854 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:45.748710    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-plugins-config-volume podName:79631c91-1914-4a7b-81ac-5b821aefbb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:15:46.748693097 +0000 UTC m=+2625.664299932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr" (UID: "79631c91-1914-4a7b-81ac-5b821aefbb6d") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-d97cc8" not found
Oct 30 20:15:45.748854 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:45.748768    6030 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d97cc8/agc-cred-52dc941f181c: secret "agc-cred-52dc941f181c" not found
Oct 30 20:15:45.748854 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:45.748792    6030 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr: secret "agc-cred-52dc941f181c" not found
Oct 30 20:15:45.748980 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:45.748865    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-provider-credentials podName:79631c91-1914-4a7b-81ac-5b821aefbb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:15:47.748848109 +0000 UTC m=+2626.664454952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr" (UID: "79631c91-1914-4a7b-81ac-5b821aefbb6d") : secret "agc-cred-52dc941f181c" not found
Oct 30 20:15:46.118655 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:46.118557    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5" podUID="c79d7c3c-99cb-44c0-88c6-23daef726eec" containerName="payload-processing" containerID="cri-o://5c0aaa4837ee862fd43bb18392cae045d57d6eaa9c722aa7bd1de5891bad051c" gracePeriod=25
Oct 30 20:15:46.404620 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:46.404591    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:15:46.459673 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:46.459638    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c79d7c3c-99cb-44c0-88c6-23daef726eec-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c79d7c3c-99cb-44c0-88c6-23daef726eec-provider-credentials\") pod \"c79d7c3c-99cb-44c0-88c6-23daef726eec\" (UID: \"c79d7c3c-99cb-44c0-88c6-23daef726eec\") "
Oct 30 20:15:46.459943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:46.459921    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c79d7c3c-99cb-44c0-88c6-23daef726eec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c79d7c3c-99cb-44c0-88c6-23daef726eec-plugins-config-volume\") pod \"c79d7c3c-99cb-44c0-88c6-23daef726eec\" (UID: \"c79d7c3c-99cb-44c0-88c6-23daef726eec\") "
Oct 30 20:15:46.460089 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:46.460032    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c79d7c3c-99cb-44c0-88c6-23daef726eec-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c79d7c3c-99cb-44c0-88c6-23daef726eec-routing-overlay-volume\") pod \"c79d7c3c-99cb-44c0-88c6-23daef726eec\" (UID: \"c79d7c3c-99cb-44c0-88c6-23daef726eec\") "
Oct 30 20:15:46.460183 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:46.460088    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c79d7c3c-99cb-44c0-88c6-23daef726eec-kube-api-access-k9vpj\" (UniqueName: \"kubernetes.io/projected/c79d7c3c-99cb-44c0-88c6-23daef726eec-kube-api-access-k9vpj\") pod \"c79d7c3c-99cb-44c0-88c6-23daef726eec\" (UID: \"c79d7c3c-99cb-44c0-88c6-23daef726eec\") "
Oct 30 20:15:46.460424 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:46.460402    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79d7c3c-99cb-44c0-88c6-23daef726eec-plugins-config-volume" pod "c79d7c3c-99cb-44c0-88c6-23daef726eec" (UID: "c79d7c3c-99cb-44c0-88c6-23daef726eec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:46.460503 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:46.460453    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79d7c3c-99cb-44c0-88c6-23daef726eec-routing-overlay-volume" pod "c79d7c3c-99cb-44c0-88c6-23daef726eec" (UID: "c79d7c3c-99cb-44c0-88c6-23daef726eec"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:46.460693 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:46.460676    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c79d7c3c-99cb-44c0-88c6-23daef726eec-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:46.460779 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:46.460695    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c79d7c3c-99cb-44c0-88c6-23daef726eec-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:46.462373 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:46.462334    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79d7c3c-99cb-44c0-88c6-23daef726eec-provider-credentials" pod "c79d7c3c-99cb-44c0-88c6-23daef726eec" (UID: "c79d7c3c-99cb-44c0-88c6-23daef726eec"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:46.462651 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:46.462631    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79d7c3c-99cb-44c0-88c6-23daef726eec-kube-api-access-k9vpj" pod "c79d7c3c-99cb-44c0-88c6-23daef726eec" (UID: "c79d7c3c-99cb-44c0-88c6-23daef726eec"). InnerVolumeSpecName "kube-api-access-k9vpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:46.561895 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:46.561865    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9vpj\" (UniqueName: \"kubernetes.io/projected/c79d7c3c-99cb-44c0-88c6-23daef726eec-kube-api-access-k9vpj\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:46.561895 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:46.561893    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c79d7c3c-99cb-44c0-88c6-23daef726eec-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:46.765958 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:46.765503    6030 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-plugins-e2e-extmod-b-d97cc8: configmap "payload-processing-external-model-plugins-e2e-extmod-b-d97cc8" not found
Oct 30 20:15:46.765958 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:46.765596    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-plugins-config-volume podName:79631c91-1914-4a7b-81ac-5b821aefbb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:15:48.765576535 +0000 UTC m=+2627.681183388 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr" (UID: "79631c91-1914-4a7b-81ac-5b821aefbb6d") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-d97cc8" not found
Oct 30 20:15:47.085408 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:47.085312    6030 generic.go:356] "Generic (PLEG): container finished" podID="c79d7c3c-99cb-44c0-88c6-23daef726eec" containerID="5c0aaa4837ee862fd43bb18392cae045d57d6eaa9c722aa7bd1de5891bad051c" exitCode=0
Oct 30 20:15:47.085408 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:47.085370    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"
Oct 30 20:15:47.086438 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:47.085400    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5" event={"ID":"c79d7c3c-99cb-44c0-88c6-23daef726eec","Type":"ContainerDied","Data":"5c0aaa4837ee862fd43bb18392cae045d57d6eaa9c722aa7bd1de5891bad051c"}
Oct 30 20:15:47.086438 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:47.085476    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5" event={"ID":"c79d7c3c-99cb-44c0-88c6-23daef726eec","Type":"ContainerDied","Data":"19575147378a918a337ad0ffbaf15f7b7da003bf62fcd892630aaa4825441a48"}
Oct 30 20:15:47.086438 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:47.085519    6030 scope.go:122] "RemoveContainer" containerID="5c0aaa4837ee862fd43bb18392cae045d57d6eaa9c722aa7bd1de5891bad051c"
Oct 30 20:15:47.147074 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:47.146129    6030 scope.go:122] "RemoveContainer" containerID="5c0aaa4837ee862fd43bb18392cae045d57d6eaa9c722aa7bd1de5891bad051c"
Oct 30 20:15:47.147074 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:47.146603    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c0aaa4837ee862fd43bb18392cae045d57d6eaa9c722aa7bd1de5891bad051c\": container with ID starting with 5c0aaa4837ee862fd43bb18392cae045d57d6eaa9c722aa7bd1de5891bad051c not found: ID does not exist" containerID="5c0aaa4837ee862fd43bb18392cae045d57d6eaa9c722aa7bd1de5891bad051c"
Oct 30 20:15:47.147074 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:47.146644    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c0aaa4837ee862fd43bb18392cae045d57d6eaa9c722aa7bd1de5891bad051c"} err="failed to get container status \"5c0aaa4837ee862fd43bb18392cae045d57d6eaa9c722aa7bd1de5891bad051c\": rpc error: code = NotFound desc = could not find container \"5c0aaa4837ee862fd43bb18392cae045d57d6eaa9c722aa7bd1de5891bad051c\": container with ID starting with 5c0aaa4837ee862fd43bb18392cae045d57d6eaa9c722aa7bd1de5891bad051c not found: ID does not exist"
Oct 30 20:15:47.245100 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:47.245062    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"]
Oct 30 20:15:47.245296 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:47.245113    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-58977ddb8f-w62d5"]
Oct 30 20:15:47.784103 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:47.784074    6030 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d97cc8/agc-cred-52dc941f181c: secret "agc-cred-52dc941f181c" not found
Oct 30 20:15:47.784103 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:47.784099    6030 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr: secret "agc-cred-52dc941f181c" not found
Oct 30 20:15:47.784347 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:47.784171    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-provider-credentials podName:79631c91-1914-4a7b-81ac-5b821aefbb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:15:51.784148145 +0000 UTC m=+2630.699754979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr" (UID: "79631c91-1914-4a7b-81ac-5b821aefbb6d") : secret "agc-cred-52dc941f181c" not found
Oct 30 20:15:48.799547 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:48.799508    6030 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-plugins-e2e-extmod-b-d97cc8: configmap "payload-processing-external-model-plugins-e2e-extmod-b-d97cc8" not found
Oct 30 20:15:48.800104 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:48.799605    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-plugins-config-volume podName:79631c91-1914-4a7b-81ac-5b821aefbb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:15:52.799585755 +0000 UTC m=+2631.715192592 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr" (UID: "79631c91-1914-4a7b-81ac-5b821aefbb6d") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-d97cc8" not found
Oct 30 20:15:49.211627 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:49.211583    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c79d7c3c-99cb-44c0-88c6-23daef726eec" path="/var/lib/kubelet/pods/c79d7c3c-99cb-44c0-88c6-23daef726eec/volumes"
Oct 30 20:15:49.598212 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:49.598108    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g" podUID="6b3c0d95-564b-4989-bc2e-0e1ad394f07f" containerName="payload-processing" containerID="cri-o://891a520be807e304689bdf9a24ad8b216fce4a4e0467ad8d15c01d7cde0190a5" gracePeriod=25
Oct 30 20:15:49.880964 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:49.880938    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:15:49.916999 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:49.916960    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-routing-overlay-volume\") pod \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\" (UID: \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\") "
Oct 30 20:15:49.917196 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:49.917165    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-provider-credentials\") pod \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\" (UID: \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\") "
Oct 30 20:15:49.917247 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:49.917235    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-kube-api-access-7d7wt\" (UniqueName: \"kubernetes.io/projected/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-kube-api-access-7d7wt\") pod \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\" (UID: \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\") "
Oct 30 20:15:49.917320 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:49.917305    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-plugins-config-volume\") pod \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\" (UID: \"6b3c0d95-564b-4989-bc2e-0e1ad394f07f\") "
Oct 30 20:15:49.917463 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:49.917417    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-routing-overlay-volume" pod "6b3c0d95-564b-4989-bc2e-0e1ad394f07f" (UID: "6b3c0d95-564b-4989-bc2e-0e1ad394f07f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:49.919157 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:49.917700    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-plugins-config-volume" pod "6b3c0d95-564b-4989-bc2e-0e1ad394f07f" (UID: "6b3c0d95-564b-4989-bc2e-0e1ad394f07f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:49.919157 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:49.918127    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:49.919157 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:49.918150    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:49.919867 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:49.919838    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-kube-api-access-7d7wt" pod "6b3c0d95-564b-4989-bc2e-0e1ad394f07f" (UID: "6b3c0d95-564b-4989-bc2e-0e1ad394f07f"). InnerVolumeSpecName "kube-api-access-7d7wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:49.920030 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:49.920007    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-provider-credentials" pod "6b3c0d95-564b-4989-bc2e-0e1ad394f07f" (UID: "6b3c0d95-564b-4989-bc2e-0e1ad394f07f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:50.019443 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.019413    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:50.019443 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.019440    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d7wt\" (UniqueName: \"kubernetes.io/projected/6b3c0d95-564b-4989-bc2e-0e1ad394f07f-kube-api-access-7d7wt\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:50.110141 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.110061    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2" podUID="0378ab28-33fd-4012-8a51-239e35ef201d" containerName="payload-processing" containerID="cri-o://9c5733d4c45cdcff091caa4b32c7a4a6ed2f3310ad4667ca4358e51e1043562a" gracePeriod=25
Oct 30 20:15:50.118137 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.118104    6030 generic.go:356] "Generic (PLEG): container finished" podID="6b3c0d95-564b-4989-bc2e-0e1ad394f07f" containerID="891a520be807e304689bdf9a24ad8b216fce4a4e0467ad8d15c01d7cde0190a5" exitCode=0
Oct 30 20:15:50.118297 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.118141    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g" event={"ID":"6b3c0d95-564b-4989-bc2e-0e1ad394f07f","Type":"ContainerDied","Data":"891a520be807e304689bdf9a24ad8b216fce4a4e0467ad8d15c01d7cde0190a5"}
Oct 30 20:15:50.118297 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.118163    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"
Oct 30 20:15:50.118297 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.118239    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g" event={"ID":"6b3c0d95-564b-4989-bc2e-0e1ad394f07f","Type":"ContainerDied","Data":"5c7182b476ef566552cf89d402f8b7ceac2f3d6e1ee8a8ba767e2d90aeffa217"}
Oct 30 20:15:50.118297 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.118273    6030 scope.go:122] "RemoveContainer" containerID="891a520be807e304689bdf9a24ad8b216fce4a4e0467ad8d15c01d7cde0190a5"
Oct 30 20:15:50.130613 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.130595    6030 scope.go:122] "RemoveContainer" containerID="891a520be807e304689bdf9a24ad8b216fce4a4e0467ad8d15c01d7cde0190a5"
Oct 30 20:15:50.130901 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:50.130884    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"891a520be807e304689bdf9a24ad8b216fce4a4e0467ad8d15c01d7cde0190a5\": container with ID starting with 891a520be807e304689bdf9a24ad8b216fce4a4e0467ad8d15c01d7cde0190a5 not found: ID does not exist" containerID="891a520be807e304689bdf9a24ad8b216fce4a4e0467ad8d15c01d7cde0190a5"
Oct 30 20:15:50.130967 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.130909    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"891a520be807e304689bdf9a24ad8b216fce4a4e0467ad8d15c01d7cde0190a5"} err="failed to get container status \"891a520be807e304689bdf9a24ad8b216fce4a4e0467ad8d15c01d7cde0190a5\": rpc error: code = NotFound desc = could not find container \"891a520be807e304689bdf9a24ad8b216fce4a4e0467ad8d15c01d7cde0190a5\": container with ID starting with 891a520be807e304689bdf9a24ad8b216fce4a4e0467ad8d15c01d7cde0190a5 not found: ID does not exist"
Oct 30 20:15:50.183099 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.183036    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"]
Oct 30 20:15:50.186177 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.186150    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5db55g86g"]
Oct 30 20:15:50.367878 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.367822    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:15:50.424957 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.424924    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0378ab28-33fd-4012-8a51-239e35ef201d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0378ab28-33fd-4012-8a51-239e35ef201d-plugins-config-volume\") pod \"0378ab28-33fd-4012-8a51-239e35ef201d\" (UID: \"0378ab28-33fd-4012-8a51-239e35ef201d\") "
Oct 30 20:15:50.425143 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.425109    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0378ab28-33fd-4012-8a51-239e35ef201d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0378ab28-33fd-4012-8a51-239e35ef201d-routing-overlay-volume\") pod \"0378ab28-33fd-4012-8a51-239e35ef201d\" (UID: \"0378ab28-33fd-4012-8a51-239e35ef201d\") "
Oct 30 20:15:50.425286 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.425270    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0378ab28-33fd-4012-8a51-239e35ef201d-kube-api-access-grmpx\" (UniqueName: \"kubernetes.io/projected/0378ab28-33fd-4012-8a51-239e35ef201d-kube-api-access-grmpx\") pod \"0378ab28-33fd-4012-8a51-239e35ef201d\" (UID: \"0378ab28-33fd-4012-8a51-239e35ef201d\") "
Oct 30 20:15:50.425395 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.425378    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0378ab28-33fd-4012-8a51-239e35ef201d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0378ab28-33fd-4012-8a51-239e35ef201d-provider-credentials\") pod \"0378ab28-33fd-4012-8a51-239e35ef201d\" (UID: \"0378ab28-33fd-4012-8a51-239e35ef201d\") "
Oct 30 20:15:50.425499 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.425406    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0378ab28-33fd-4012-8a51-239e35ef201d-plugins-config-volume" pod "0378ab28-33fd-4012-8a51-239e35ef201d" (UID: "0378ab28-33fd-4012-8a51-239e35ef201d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:50.425499 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.425458    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0378ab28-33fd-4012-8a51-239e35ef201d-routing-overlay-volume" pod "0378ab28-33fd-4012-8a51-239e35ef201d" (UID: "0378ab28-33fd-4012-8a51-239e35ef201d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:50.426209 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.426112    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0378ab28-33fd-4012-8a51-239e35ef201d-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:50.426334 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.426210    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0378ab28-33fd-4012-8a51-239e35ef201d-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:50.427668 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.427638    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0378ab28-33fd-4012-8a51-239e35ef201d-kube-api-access-grmpx" pod "0378ab28-33fd-4012-8a51-239e35ef201d" (UID: "0378ab28-33fd-4012-8a51-239e35ef201d"). InnerVolumeSpecName "kube-api-access-grmpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:50.427750 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.427714    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0378ab28-33fd-4012-8a51-239e35ef201d-provider-credentials" pod "0378ab28-33fd-4012-8a51-239e35ef201d" (UID: "0378ab28-33fd-4012-8a51-239e35ef201d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:50.527351 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.527306    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grmpx\" (UniqueName: \"kubernetes.io/projected/0378ab28-33fd-4012-8a51-239e35ef201d-kube-api-access-grmpx\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:50.527351 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:50.527341    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0378ab28-33fd-4012-8a51-239e35ef201d-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:51.074553 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.074450    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr" podUID="79631c91-1914-4a7b-81ac-5b821aefbb6d" containerName="payload-processing" containerID="cri-o://c7a606b9d31ca3454324a821532ff664d1310b30293f65a220899cd0026ae529" gracePeriod=25
Oct 30 20:15:51.128440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.128410    6030 generic.go:356] "Generic (PLEG): container finished" podID="0378ab28-33fd-4012-8a51-239e35ef201d" containerID="9c5733d4c45cdcff091caa4b32c7a4a6ed2f3310ad4667ca4358e51e1043562a" exitCode=0
Oct 30 20:15:51.128551 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.128473    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"
Oct 30 20:15:51.128551 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.128501    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2" event={"ID":"0378ab28-33fd-4012-8a51-239e35ef201d","Type":"ContainerDied","Data":"9c5733d4c45cdcff091caa4b32c7a4a6ed2f3310ad4667ca4358e51e1043562a"}
Oct 30 20:15:51.128635 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.128592    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2" event={"ID":"0378ab28-33fd-4012-8a51-239e35ef201d","Type":"ContainerDied","Data":"05617c22ccbae17f51396c097a4566374e92817c2cae83d1a1c9bb4c46a87942"}
Oct 30 20:15:51.128635 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.128622    6030 scope.go:122] "RemoveContainer" containerID="9c5733d4c45cdcff091caa4b32c7a4a6ed2f3310ad4667ca4358e51e1043562a"
Oct 30 20:15:51.168421 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.168396    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"]
Oct 30 20:15:51.176101 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.176079    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-79f5vdsf2"]
Oct 30 20:15:51.188106 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.188084    6030 scope.go:122] "RemoveContainer" containerID="9c5733d4c45cdcff091caa4b32c7a4a6ed2f3310ad4667ca4358e51e1043562a"
Oct 30 20:15:51.188437 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:51.188410    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5733d4c45cdcff091caa4b32c7a4a6ed2f3310ad4667ca4358e51e1043562a\": container with ID starting with 9c5733d4c45cdcff091caa4b32c7a4a6ed2f3310ad4667ca4358e51e1043562a not found: ID does not exist" containerID="9c5733d4c45cdcff091caa4b32c7a4a6ed2f3310ad4667ca4358e51e1043562a"
Oct 30 20:15:51.188502 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.188450    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5733d4c45cdcff091caa4b32c7a4a6ed2f3310ad4667ca4358e51e1043562a"} err="failed to get container status \"9c5733d4c45cdcff091caa4b32c7a4a6ed2f3310ad4667ca4358e51e1043562a\": rpc error: code = NotFound desc = could not find container \"9c5733d4c45cdcff091caa4b32c7a4a6ed2f3310ad4667ca4358e51e1043562a\": container with ID starting with 9c5733d4c45cdcff091caa4b32c7a4a6ed2f3310ad4667ca4358e51e1043562a not found: ID does not exist"
Oct 30 20:15:51.203645 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.203622    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0378ab28-33fd-4012-8a51-239e35ef201d" path="/var/lib/kubelet/pods/0378ab28-33fd-4012-8a51-239e35ef201d/volumes"
Oct 30 20:15:51.204320 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.204297    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3c0d95-564b-4989-bc2e-0e1ad394f07f" path="/var/lib/kubelet/pods/6b3c0d95-564b-4989-bc2e-0e1ad394f07f/volumes"
Oct 30 20:15:51.373756 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.373722    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"
Oct 30 20:15:51.439419 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.439389    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-provider-credentials\") pod \"79631c91-1914-4a7b-81ac-5b821aefbb6d\" (UID: \"79631c91-1914-4a7b-81ac-5b821aefbb6d\") "
Oct 30 20:15:51.439588 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.439558    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-plugins-config-volume\") pod \"79631c91-1914-4a7b-81ac-5b821aefbb6d\" (UID: \"79631c91-1914-4a7b-81ac-5b821aefbb6d\") "
Oct 30 20:15:51.439694 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.439616    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-routing-overlay-volume\") pod \"79631c91-1914-4a7b-81ac-5b821aefbb6d\" (UID: \"79631c91-1914-4a7b-81ac-5b821aefbb6d\") "
Oct 30 20:15:51.439694 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.439677    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-kube-api-access-z47qw\" (UniqueName: \"kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-kube-api-access-z47qw\") pod \"79631c91-1914-4a7b-81ac-5b821aefbb6d\" (UID: \"79631c91-1914-4a7b-81ac-5b821aefbb6d\") "
Oct 30 20:15:51.440183 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.440156    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-plugins-config-volume" pod "79631c91-1914-4a7b-81ac-5b821aefbb6d" (UID: "79631c91-1914-4a7b-81ac-5b821aefbb6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:51.440477 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.440450    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-routing-overlay-volume" pod "79631c91-1914-4a7b-81ac-5b821aefbb6d" (UID: "79631c91-1914-4a7b-81ac-5b821aefbb6d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:51.442634 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.442604    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-provider-credentials" pod "79631c91-1914-4a7b-81ac-5b821aefbb6d" (UID: "79631c91-1914-4a7b-81ac-5b821aefbb6d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:51.442745 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.442706    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-kube-api-access-z47qw" pod "79631c91-1914-4a7b-81ac-5b821aefbb6d" (UID: "79631c91-1914-4a7b-81ac-5b821aefbb6d"). InnerVolumeSpecName "kube-api-access-z47qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:51.541085 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.541026    6030 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-provider-credentials\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:51.541217 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.541089    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:51.541217 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.541104    6030 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79631c91-1914-4a7b-81ac-5b821aefbb6d-routing-overlay-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:51.541217 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.541116    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z47qw\" (UniqueName: \"kubernetes.io/projected/79631c91-1914-4a7b-81ac-5b821aefbb6d-kube-api-access-z47qw\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:51.925101 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.910570    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"]
Oct 30 20:15:51.925101 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:51.920265    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"]
Oct 30 20:15:52.138337 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.138307    6030 generic.go:356] "Generic (PLEG): container finished" podID="79631c91-1914-4a7b-81ac-5b821aefbb6d" containerID="c7a606b9d31ca3454324a821532ff664d1310b30293f65a220899cd0026ae529" exitCode=0
Oct 30 20:15:52.138782 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.138353    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"
Oct 30 20:15:52.138782 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.138388    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr" event={"ID":"79631c91-1914-4a7b-81ac-5b821aefbb6d","Type":"ContainerDied","Data":"c7a606b9d31ca3454324a821532ff664d1310b30293f65a220899cd0026ae529"}
Oct 30 20:15:52.138782 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.138440    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr" event={"ID":"79631c91-1914-4a7b-81ac-5b821aefbb6d","Type":"ContainerDied","Data":"2a9e60fcdb353e24584566bb4e124b1095058cbb4cd98acf54201c352e20c612"}
Oct 30 20:15:52.138782 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.138467    6030 scope.go:122] "RemoveContainer" containerID="c7a606b9d31ca3454324a821532ff664d1310b30293f65a220899cd0026ae529"
Oct 30 20:15:52.149781 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.149762    6030 scope.go:122] "RemoveContainer" containerID="c7a606b9d31ca3454324a821532ff664d1310b30293f65a220899cd0026ae529"
Oct 30 20:15:52.150098 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:52.150068    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a606b9d31ca3454324a821532ff664d1310b30293f65a220899cd0026ae529\": container with ID starting with c7a606b9d31ca3454324a821532ff664d1310b30293f65a220899cd0026ae529 not found: ID does not exist" containerID="c7a606b9d31ca3454324a821532ff664d1310b30293f65a220899cd0026ae529"
Oct 30 20:15:52.150263 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.150107    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a606b9d31ca3454324a821532ff664d1310b30293f65a220899cd0026ae529"} err="failed to get container status \"c7a606b9d31ca3454324a821532ff664d1310b30293f65a220899cd0026ae529\": rpc error: code = NotFound desc = could not find container \"c7a606b9d31ca3454324a821532ff664d1310b30293f65a220899cd0026ae529\": container with ID starting with c7a606b9d31ca3454324a821532ff664d1310b30293f65a220899cd0026ae529 not found: ID does not exist"
Oct 30 20:15:52.180429 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.180394    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"]
Oct 30 20:15:52.187850 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.187826    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6645kbvvr"]
Oct 30 20:15:52.357419 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.357375    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"]
Oct 30 20:15:52.357922 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.357885    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt" podUID="f68b3878-c377-4994-9b24-1b2090038ba3" containerName="maas-api" containerID="cri-o://6f3ce1197f17ad5bbeeabe272607f3036d3e715117d1064c8d549b082e648de9" gracePeriod=30
Oct 30 20:15:52.667549 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.667472    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:15:52.761706 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.761676    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f68b3878-c377-4994-9b24-1b2090038ba3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68b3878-c377-4994-9b24-1b2090038ba3-openshift-service-ca\") pod \"f68b3878-c377-4994-9b24-1b2090038ba3\" (UID: \"f68b3878-c377-4994-9b24-1b2090038ba3\") "
Oct 30 20:15:52.761908 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.761844    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f68b3878-c377-4994-9b24-1b2090038ba3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f68b3878-c377-4994-9b24-1b2090038ba3-maas-api-tls\") pod \"f68b3878-c377-4994-9b24-1b2090038ba3\" (UID: \"f68b3878-c377-4994-9b24-1b2090038ba3\") "
Oct 30 20:15:52.761988 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.761925    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f68b3878-c377-4994-9b24-1b2090038ba3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f68b3878-c377-4994-9b24-1b2090038ba3-metrics-certs\") pod \"f68b3878-c377-4994-9b24-1b2090038ba3\" (UID: \"f68b3878-c377-4994-9b24-1b2090038ba3\") "
Oct 30 20:15:52.762062 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.761992    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f68b3878-c377-4994-9b24-1b2090038ba3-kube-api-access-jzxn2\" (UniqueName: \"kubernetes.io/projected/f68b3878-c377-4994-9b24-1b2090038ba3-kube-api-access-jzxn2\") pod \"f68b3878-c377-4994-9b24-1b2090038ba3\" (UID: \"f68b3878-c377-4994-9b24-1b2090038ba3\") "
Oct 30 20:15:52.762134 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.762102    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68b3878-c377-4994-9b24-1b2090038ba3-openshift-service-ca" pod "f68b3878-c377-4994-9b24-1b2090038ba3" (UID: "f68b3878-c377-4994-9b24-1b2090038ba3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:52.762473 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.762451    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68b3878-c377-4994-9b24-1b2090038ba3-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:52.764444 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.764413    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68b3878-c377-4994-9b24-1b2090038ba3-kube-api-access-jzxn2" pod "f68b3878-c377-4994-9b24-1b2090038ba3" (UID: "f68b3878-c377-4994-9b24-1b2090038ba3"). InnerVolumeSpecName "kube-api-access-jzxn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:52.764729 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.764710    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f68b3878-c377-4994-9b24-1b2090038ba3-metrics-certs" pod "f68b3878-c377-4994-9b24-1b2090038ba3" (UID: "f68b3878-c377-4994-9b24-1b2090038ba3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:52.764796 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.764732    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f68b3878-c377-4994-9b24-1b2090038ba3-maas-api-tls" pod "f68b3878-c377-4994-9b24-1b2090038ba3" (UID: "f68b3878-c377-4994-9b24-1b2090038ba3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:52.863754 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.863716    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f68b3878-c377-4994-9b24-1b2090038ba3-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:52.863754 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.863746    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f68b3878-c377-4994-9b24-1b2090038ba3-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:52.863754 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:52.863762    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzxn2\" (UniqueName: \"kubernetes.io/projected/f68b3878-c377-4994-9b24-1b2090038ba3-kube-api-access-jzxn2\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:53.153315 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:53.153280    6030 generic.go:356] "Generic (PLEG): container finished" podID="f68b3878-c377-4994-9b24-1b2090038ba3" containerID="6f3ce1197f17ad5bbeeabe272607f3036d3e715117d1064c8d549b082e648de9" exitCode=0
Oct 30 20:15:53.153786 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:53.153334    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"
Oct 30 20:15:53.153786 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:53.153361    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt" event={"ID":"f68b3878-c377-4994-9b24-1b2090038ba3","Type":"ContainerDied","Data":"6f3ce1197f17ad5bbeeabe272607f3036d3e715117d1064c8d549b082e648de9"}
Oct 30 20:15:53.153786 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:53.153431    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt" event={"ID":"f68b3878-c377-4994-9b24-1b2090038ba3","Type":"ContainerDied","Data":"89e44d042dda6a8ed5da399149aa8de71991cfedb1ab6cef41d3109610ec68dd"}
Oct 30 20:15:53.153786 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:53.153469    6030 scope.go:122] "RemoveContainer" containerID="6f3ce1197f17ad5bbeeabe272607f3036d3e715117d1064c8d549b082e648de9"
Oct 30 20:15:53.165445 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:53.165427    6030 scope.go:122] "RemoveContainer" containerID="6f3ce1197f17ad5bbeeabe272607f3036d3e715117d1064c8d549b082e648de9"
Oct 30 20:15:53.165720 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:15:53.165701    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f3ce1197f17ad5bbeeabe272607f3036d3e715117d1064c8d549b082e648de9\": container with ID starting with 6f3ce1197f17ad5bbeeabe272607f3036d3e715117d1064c8d549b082e648de9 not found: ID does not exist" containerID="6f3ce1197f17ad5bbeeabe272607f3036d3e715117d1064c8d549b082e648de9"
Oct 30 20:15:53.165794 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:53.165733    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f3ce1197f17ad5bbeeabe272607f3036d3e715117d1064c8d549b082e648de9"} err="failed to get container status \"6f3ce1197f17ad5bbeeabe272607f3036d3e715117d1064c8d549b082e648de9\": rpc error: code = NotFound desc = could not find container \"6f3ce1197f17ad5bbeeabe272607f3036d3e715117d1064c8d549b082e648de9\": container with ID starting with 6f3ce1197f17ad5bbeeabe272607f3036d3e715117d1064c8d549b082e648de9 not found: ID does not exist"
Oct 30 20:15:53.185014 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:53.184983    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"]
Oct 30 20:15:53.193921 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:53.190540    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-68685b-6fc846f74-pj4pt"]
Oct 30 20:15:53.211891 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:53.211863    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79631c91-1914-4a7b-81ac-5b821aefbb6d" path="/var/lib/kubelet/pods/79631c91-1914-4a7b-81ac-5b821aefbb6d/volumes"
Oct 30 20:15:53.212628 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:53.212607    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f68b3878-c377-4994-9b24-1b2090038ba3" path="/var/lib/kubelet/pods/f68b3878-c377-4994-9b24-1b2090038ba3/volumes"
Oct 30 20:15:54.352503 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.352460    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"]
Oct 30 20:15:54.353524 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353500    6030 cpu_manager.go:414] "containerMap: removing container" podUID="f68b3878-c377-4994-9b24-1b2090038ba3" containerName="maas-api"
Oct 30 20:15:54.353524 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353521    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68b3878-c377-4994-9b24-1b2090038ba3" containerName="maas-api"
Oct 30 20:15:54.353713 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353561    6030 cpu_manager.go:414] "containerMap: removing container" podUID="6b3c0d95-564b-4989-bc2e-0e1ad394f07f" containerName="payload-processing"
Oct 30 20:15:54.353713 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353569    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3c0d95-564b-4989-bc2e-0e1ad394f07f" containerName="payload-processing"
Oct 30 20:15:54.353713 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353612    6030 cpu_manager.go:414] "containerMap: removing container" podUID="79631c91-1914-4a7b-81ac-5b821aefbb6d" containerName="payload-processing"
Oct 30 20:15:54.353713 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353621    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79631c91-1914-4a7b-81ac-5b821aefbb6d" containerName="payload-processing"
Oct 30 20:15:54.353713 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353645    6030 cpu_manager.go:414] "containerMap: removing container" podUID="0378ab28-33fd-4012-8a51-239e35ef201d" containerName="payload-processing"
Oct 30 20:15:54.353713 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353653    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0378ab28-33fd-4012-8a51-239e35ef201d" containerName="payload-processing"
Oct 30 20:15:54.353713 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353668    6030 cpu_manager.go:414] "containerMap: removing container" podUID="c79d7c3c-99cb-44c0-88c6-23daef726eec" containerName="payload-processing"
Oct 30 20:15:54.353713 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353676    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c79d7c3c-99cb-44c0-88c6-23daef726eec" containerName="payload-processing"
Oct 30 20:15:54.353713 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353692    6030 cpu_manager.go:414] "containerMap: removing container" podUID="9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b" containerName="cleanup"
Oct 30 20:15:54.353713 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353700    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cbc3db5-b092-4b2a-8c94-baf0afa8ce5b" containerName="cleanup"
Oct 30 20:15:54.354273 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353851    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68b3878-c377-4994-9b24-1b2090038ba3" containerName="maas-api"
Oct 30 20:15:54.354273 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353871    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3c0d95-564b-4989-bc2e-0e1ad394f07f" containerName="payload-processing"
Oct 30 20:15:54.354273 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353887    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="79631c91-1914-4a7b-81ac-5b821aefbb6d" containerName="payload-processing"
Oct 30 20:15:54.354273 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353898    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="461521b6-3031-46d9-8e54-765c46db9983" containerName="cleanup"
Oct 30 20:15:54.354273 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353908    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="0378ab28-33fd-4012-8a51-239e35ef201d" containerName="payload-processing"
Oct 30 20:15:54.354273 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353916    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="c79d7c3c-99cb-44c0-88c6-23daef726eec" containerName="payload-processing"
Oct 30 20:15:54.354273 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.353927    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="380956ae-205c-4814-aa2d-68aae67dd2cc" containerName="cleanup"
Oct 30 20:15:54.359255 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.359231    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.369511 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.369490    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c257e7-openshift-default-dockercfg-rxfv5\""
Oct 30 20:15:54.375190 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.375162    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"]
Oct 30 20:15:54.484581 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.484544    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6484295e-f0bf-4194-9416-12c15129dd4a-istiod-ca-cert\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.484954 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.484920    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6484295e-f0bf-4194-9416-12c15129dd4a-istio-podinfo\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.485155 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.485135    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-istio-envoy\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.485257 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.485195    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfn96\" (UniqueName: \"kubernetes.io/projected/6484295e-f0bf-4194-9416-12c15129dd4a-kube-api-access-qfn96\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.485257 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.485255    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-credential-socket\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.485376 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.485276    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-workload-socket\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.485376 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.485298    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-istio-data\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.485376 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.485325    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6484295e-f0bf-4194-9416-12c15129dd4a-istio-token\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.485515 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.485479    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-workload-certs\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.588941 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.588890    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-workload-certs\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.589193 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.588976    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6484295e-f0bf-4194-9416-12c15129dd4a-istiod-ca-cert\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.589193 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.589019    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6484295e-f0bf-4194-9416-12c15129dd4a-istio-podinfo\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.589193 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.589170    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-istio-envoy\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.589390 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.589223    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfn96\" (UniqueName: \"kubernetes.io/projected/6484295e-f0bf-4194-9416-12c15129dd4a-kube-api-access-qfn96\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.589390 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.589302    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-credential-socket\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.589390 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.589333    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-workload-socket\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.589390 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.589363    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-istio-data\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.589616 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.589393    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-workload-certs\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.589616 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.589407    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6484295e-f0bf-4194-9416-12c15129dd4a-istio-token\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.589878 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.589854    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6484295e-f0bf-4194-9416-12c15129dd4a-istiod-ca-cert\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.590011 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.589987    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-credential-socket\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.590149 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.590127    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-workload-socket\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.590252 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.590180    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-istio-data\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.595786 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.593550    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6484295e-f0bf-4194-9416-12c15129dd4a-istio-podinfo\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.597232 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.596238    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-istio-envoy\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.616977 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.616800    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6484295e-f0bf-4194-9416-12c15129dd4a-istio-token\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.617856 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.617832    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfn96\" (UniqueName: \"kubernetes.io/projected/6484295e-f0bf-4194-9416-12c15129dd4a-kube-api-access-qfn96\") pod \"e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") " pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:54.679091 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:54.679034    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:55.023289 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:15:55.023260    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6484295e_f0bf_4194_9416_12c15129dd4a.slice/crio-9c2a6af9116bacdbbe46f8d2fb83092d9bd61fdbcc79aa51ac1dc375547974ad WatchSource:0}: Error finding container 9c2a6af9116bacdbbe46f8d2fb83092d9bd61fdbcc79aa51ac1dc375547974ad: Status 404 returned error can't find the container with id 9c2a6af9116bacdbbe46f8d2fb83092d9bd61fdbcc79aa51ac1dc375547974ad
Oct 30 20:15:55.024279 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:55.024145    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"]
Oct 30 20:15:55.027132 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:55.027091    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:15:55.027233 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:55.027181    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:15:55.027287 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:55.027231    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:15:55.175927 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:55.175886    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2" event={"ID":"6484295e-f0bf-4194-9416-12c15129dd4a","Type":"ContainerStarted","Data":"9c46eb3acfb2f2da869e5cd6f8096318a029748e3ee24ea055bcdeb0cb05cccf"}
Oct 30 20:15:55.176159 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:55.175952    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2" event={"ID":"6484295e-f0bf-4194-9416-12c15129dd4a","Type":"ContainerStarted","Data":"9c2a6af9116bacdbbe46f8d2fb83092d9bd61fdbcc79aa51ac1dc375547974ad"}
Oct 30 20:15:55.679960 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:55.679926    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:56.686762 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:56.686724    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:56.718311 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:56.718262    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2" podStartSLOduration=2.7182475630000003 podStartE2EDuration="2.718247563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:55.207335418 +0000 UTC m=+2634.122942275" watchObservedRunningTime="2026-10-30 20:15:56.718247563 +0000 UTC m=+2635.633854421"
Oct 30 20:15:56.911818 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:56.911776    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27" podUID="64ec049e-096e-4146-8ad7-4d6a7398438f" containerName="payload-processing" containerID="cri-o://0d9b290fbc186d166112c3928171b2c115ae258d02279a36d310caa9d4b75e88" gracePeriod=25
Oct 30 20:15:56.921402 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:56.921352    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj" podUID="b13ed493-c629-4370-9810-7dc11c243f7f" containerName="payload-pre-processing" containerID="cri-o://1526ae99c146c30697ae1ef3d53feb81fd9965c28353db880b941eb6dc9cd0c4" gracePeriod=25
Oct 30 20:15:57.190884 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:57.190844    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:57.191847 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:57.191823    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:15:58.489022 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:58.487259    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"]
Oct 30 20:15:59.007872 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:59.007818    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj" podUID="b13ed493-c629-4370-9810-7dc11c243f7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:15:59.008091 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:59.007951    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27" podUID="64ec049e-096e-4146-8ad7-4d6a7398438f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:15:59.251643 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:59.251609    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"]
Oct 30 20:15:59.256284 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:59.256263    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"
Oct 30 20:15:59.266660 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:59.266638    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"]
Oct 30 20:15:59.315273 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:59.315243    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c257e7\""
Oct 30 20:15:59.323352 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:59.323327    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c257e7-dockercfg-hsmh4\""
Oct 30 20:15:59.367224 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:59.367183    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vstlw\" (UniqueName: \"kubernetes.io/projected/fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0-kube-api-access-vstlw\") pod \"payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt\" (UID: \"fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"
Oct 30 20:15:59.378235 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:59.378200    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt\" (UID: \"fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"
Oct 30 20:15:59.480648 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:59.480610    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vstlw\" (UniqueName: \"kubernetes.io/projected/fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0-kube-api-access-vstlw\") pod \"payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt\" (UID: \"fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"
Oct 30 20:15:59.480854 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:59.480687    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt\" (UID: \"fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"
Oct 30 20:15:59.481455 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:59.481433    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt\" (UID: \"fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"
Oct 30 20:15:59.498102 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:59.498070    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vstlw\" (UniqueName: \"kubernetes.io/projected/fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0-kube-api-access-vstlw\") pod \"payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt\" (UID: \"fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"
Oct 30 20:15:59.574596 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:59.574502    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"
Oct 30 20:15:59.874937 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:15:59.874903    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"]
Oct 30 20:15:59.876093 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:15:59.876014    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcdc56e3_b6f7_4d5a_bdbd_6e7caca428a0.slice/crio-86ffd51c35b7482c9a3fe93999f13322196ff1fb0f8816a1b0332d84f2b3093f WatchSource:0}: Error finding container 86ffd51c35b7482c9a3fe93999f13322196ff1fb0f8816a1b0332d84f2b3093f: Status 404 returned error can't find the container with id 86ffd51c35b7482c9a3fe93999f13322196ff1fb0f8816a1b0332d84f2b3093f
Oct 30 20:16:00.217111 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:00.217076    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt" event={"ID":"fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0","Type":"ContainerStarted","Data":"69e29ab39d57e55a2244aff106d8ec18521f9ea00d7bd6e1cd45362aa9931cb8"}
Oct 30 20:16:00.217317 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:00.217136    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt" event={"ID":"fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0","Type":"ContainerStarted","Data":"86ffd51c35b7482c9a3fe93999f13322196ff1fb0f8816a1b0332d84f2b3093f"}
Oct 30 20:16:00.217317 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:00.217245    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"
Oct 30 20:16:00.239481 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:00.239427    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt" podStartSLOduration=1.239412114 podStartE2EDuration="1.239412114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:00.237433392 +0000 UTC m=+2639.153040247" watchObservedRunningTime="2026-10-30 20:16:00.239412114 +0000 UTC m=+2639.155018969"
Oct 30 20:16:02.101910 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:02.101878    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"]
Oct 30 20:16:02.102434 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:02.102131    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq" podUID="d4849f63-03b7-4090-bbc9-c0039ea48195" containerName="istio-proxy" containerID="cri-o://9600c4536bf04f200156d0c8828e879eaac5d01f6790290cf80fce931e410a35" gracePeriod=30
Oct 30 20:16:03.488110 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:03.488064    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp" podUID="75ca4407-7903-475d-8c49-2dde2ba111b0" containerName="payload-pre-processing" containerID="cri-o://711d4b63f87932b08d60f4e961aa68172822d98b2ae6437f31fb22d53d4f150f" gracePeriod=25
Oct 30 20:16:04.307404 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:16:04.307367    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:16:04.578981 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:04.578896    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"]
Oct 30 20:16:04.584062 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:04.584012    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"]
Oct 30 20:16:04.894076 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:04.884205    6030 patch_prober.go:29] interesting pod/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:16:04.894076 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:04.884277    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq" podUID="d4849f63-03b7-4090-bbc9-c0039ea48195" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:16:06.303190 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:06.303157    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp" podUID="75ca4407-7903-475d-8c49-2dde2ba111b0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:07.305740 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.305711    6030 generic.go:356] "Generic (PLEG): container finished" podID="d4849f63-03b7-4090-bbc9-c0039ea48195" containerID="9600c4536bf04f200156d0c8828e879eaac5d01f6790290cf80fce931e410a35" exitCode=0
Oct 30 20:16:07.306487 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.305887    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq" event={"ID":"d4849f63-03b7-4090-bbc9-c0039ea48195","Type":"ContainerDied","Data":"9600c4536bf04f200156d0c8828e879eaac5d01f6790290cf80fce931e410a35"}
Oct 30 20:16:07.306603 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.306572    6030 scope.go:122] "RemoveContainer" containerID="ddbfcf9a0e65438f3149dc7368c25c4289329c9ddff73b0c7c61c76af5e76a73"
Oct 30 20:16:07.383553 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.383520    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:16:07.510098 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.510062    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-podinfo\") pod \"d4849f63-03b7-4090-bbc9-c0039ea48195\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") "
Oct 30 20:16:07.510272 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.510176    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-workload-socket\") pod \"d4849f63-03b7-4090-bbc9-c0039ea48195\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") "
Oct 30 20:16:07.510272 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.510209    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-workload-certs\") pod \"d4849f63-03b7-4090-bbc9-c0039ea48195\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") "
Oct 30 20:16:07.510390 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.510333    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-token\" (UniqueName: \"kubernetes.io/projected/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-token\") pod \"d4849f63-03b7-4090-bbc9-c0039ea48195\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") "
Oct 30 20:16:07.510452 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.510419    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4849f63-03b7-4090-bbc9-c0039ea48195-kube-api-access-49k9x\" (UniqueName: \"kubernetes.io/projected/d4849f63-03b7-4090-bbc9-c0039ea48195-kube-api-access-49k9x\") pod \"d4849f63-03b7-4090-bbc9-c0039ea48195\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") "
Oct 30 20:16:07.510522 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.510458    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-workload-socket" pod "d4849f63-03b7-4090-bbc9-c0039ea48195" (UID: "d4849f63-03b7-4090-bbc9-c0039ea48195"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:07.510583 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.510483    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-envoy\") pod \"d4849f63-03b7-4090-bbc9-c0039ea48195\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") "
Oct 30 20:16:07.510583 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.510552    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-workload-certs" pod "d4849f63-03b7-4090-bbc9-c0039ea48195" (UID: "d4849f63-03b7-4090-bbc9-c0039ea48195"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:07.510665 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.510638    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4849f63-03b7-4090-bbc9-c0039ea48195-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4849f63-03b7-4090-bbc9-c0039ea48195-istiod-ca-cert\") pod \"d4849f63-03b7-4090-bbc9-c0039ea48195\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") "
Oct 30 20:16:07.510924 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.510902    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-data\") pod \"d4849f63-03b7-4090-bbc9-c0039ea48195\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") "
Oct 30 20:16:07.511165 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.510945    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-credential-socket\") pod \"d4849f63-03b7-4090-bbc9-c0039ea48195\" (UID: \"d4849f63-03b7-4090-bbc9-c0039ea48195\") "
Oct 30 20:16:07.511165 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.510966    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4849f63-03b7-4090-bbc9-c0039ea48195-istiod-ca-cert" pod "d4849f63-03b7-4090-bbc9-c0039ea48195" (UID: "d4849f63-03b7-4090-bbc9-c0039ea48195"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:07.511382 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.511301    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-data" pod "d4849f63-03b7-4090-bbc9-c0039ea48195" (UID: "d4849f63-03b7-4090-bbc9-c0039ea48195"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:07.511382 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.511315    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-credential-socket" pod "d4849f63-03b7-4090-bbc9-c0039ea48195" (UID: "d4849f63-03b7-4090-bbc9-c0039ea48195"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:07.511826 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.511800    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4849f63-03b7-4090-bbc9-c0039ea48195-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:07.511919 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.511831    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:07.511919 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.511848    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:07.511919 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.511862    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:07.511919 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.511879    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:07.512419 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.512399    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-podinfo" pod "d4849f63-03b7-4090-bbc9-c0039ea48195" (UID: "d4849f63-03b7-4090-bbc9-c0039ea48195"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:16:07.512777 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.512756    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-envoy" pod "d4849f63-03b7-4090-bbc9-c0039ea48195" (UID: "d4849f63-03b7-4090-bbc9-c0039ea48195"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:07.512886 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.512865    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4849f63-03b7-4090-bbc9-c0039ea48195-kube-api-access-49k9x" pod "d4849f63-03b7-4090-bbc9-c0039ea48195" (UID: "d4849f63-03b7-4090-bbc9-c0039ea48195"). InnerVolumeSpecName "kube-api-access-49k9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:07.512932 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.512883    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-token" pod "d4849f63-03b7-4090-bbc9-c0039ea48195" (UID: "d4849f63-03b7-4090-bbc9-c0039ea48195"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:07.612999 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.612970    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:07.612999 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.612998    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:07.613204 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.613007    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49k9x\" (UniqueName: \"kubernetes.io/projected/d4849f63-03b7-4090-bbc9-c0039ea48195-kube-api-access-49k9x\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:07.613204 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:07.613018    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4849f63-03b7-4090-bbc9-c0039ea48195-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:08.314213 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:08.314180    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"
Oct 30 20:16:08.314699 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:08.314180    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq" event={"ID":"d4849f63-03b7-4090-bbc9-c0039ea48195","Type":"ContainerDied","Data":"893242d94606143eb285c1df331376be40423d08d4478d27d02b6dd52e3d377d"}
Oct 30 20:16:08.314699 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:08.314344    6030 scope.go:122] "RemoveContainer" containerID="9600c4536bf04f200156d0c8828e879eaac5d01f6790290cf80fce931e410a35"
Oct 30 20:16:08.342795 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:08.342774    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"]
Oct 30 20:16:08.345799 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:08.345774    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-68685b-openshift-default-dd8c4cb8f-z5sgq"]
Oct 30 20:16:09.008728 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:09.008695    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj" podUID="b13ed493-c629-4370-9810-7dc11c243f7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:09.009087 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:09.008772    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27" podUID="64ec049e-096e-4146-8ad7-4d6a7398438f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:09.203580 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:09.203550    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4849f63-03b7-4090-bbc9-c0039ea48195" path="/var/lib/kubelet/pods/d4849f63-03b7-4090-bbc9-c0039ea48195/volumes"
Oct 30 20:16:09.563431 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:09.563394    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"]
Oct 30 20:16:09.563954 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:09.563643    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477" podUID="c3a85e15-16f9-4e1f-aa08-f1437a0bb44f" containerName="maas-api" containerID="cri-o://c631125ab62e34e8bf2d233d0f14d8ae6a176086b05b749fb036762ff94c4832" gracePeriod=30
Oct 30 20:16:09.580113 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:09.580073    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg" podUID="aff9109e-c1a8-4ba0-9510-dc8d18be9547" containerName="payload-processing" containerID="cri-o://a4e877eed84af948eba7b99ae2c76f8651a42ef07b86664a116c3dd80cca05fc" gracePeriod=25
Oct 30 20:16:09.583614 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:09.583582    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477" podUID="c3a85e15-16f9-4e1f-aa08-f1437a0bb44f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.159:8443/health\": dial tcp 10.133.0.159:8443: connect: connection refused"
Oct 30 20:16:09.596373 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:09.596340    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b" podUID="2c8b6e10-33d3-479c-b3ae-c43d2058022e" containerName="payload-pre-processing" containerID="cri-o://33369a845a4b39dea21c8a43bc226c714fefebd69f16a210ed69dff2af6379f0" gracePeriod=25
Oct 30 20:16:09.972527 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:09.972495    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:16:10.052844 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.052809    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-openshift-service-ca\") pod \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\" (UID: \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\") "
Oct 30 20:16:10.053039 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.052918    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-metrics-certs\") pod \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\" (UID: \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\") "
Oct 30 20:16:10.053129 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.053071    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-kube-api-access-qmhwn\" (UniqueName: \"kubernetes.io/projected/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-kube-api-access-qmhwn\") pod \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\" (UID: \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\") "
Oct 30 20:16:10.053129 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.053103    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-maas-api-tls\") pod \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\" (UID: \"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f\") "
Oct 30 20:16:10.053260 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.053153    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-openshift-service-ca" pod "c3a85e15-16f9-4e1f-aa08-f1437a0bb44f" (UID: "c3a85e15-16f9-4e1f-aa08-f1437a0bb44f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:10.053651 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.053570    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:10.055272 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.055236    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-metrics-certs" pod "c3a85e15-16f9-4e1f-aa08-f1437a0bb44f" (UID: "c3a85e15-16f9-4e1f-aa08-f1437a0bb44f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:10.055413 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.055394    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-maas-api-tls" pod "c3a85e15-16f9-4e1f-aa08-f1437a0bb44f" (UID: "c3a85e15-16f9-4e1f-aa08-f1437a0bb44f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:10.055460 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.055422    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-kube-api-access-qmhwn" pod "c3a85e15-16f9-4e1f-aa08-f1437a0bb44f" (UID: "c3a85e15-16f9-4e1f-aa08-f1437a0bb44f"). InnerVolumeSpecName "kube-api-access-qmhwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:10.155195 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.155161    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmhwn\" (UniqueName: \"kubernetes.io/projected/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-kube-api-access-qmhwn\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:10.155195 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.155195    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:10.155418 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.155208    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:10.330600 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.330514    6030 generic.go:356] "Generic (PLEG): container finished" podID="c3a85e15-16f9-4e1f-aa08-f1437a0bb44f" containerID="c631125ab62e34e8bf2d233d0f14d8ae6a176086b05b749fb036762ff94c4832" exitCode=0
Oct 30 20:16:10.330600 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.330564    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"
Oct 30 20:16:10.330600 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.330586    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477" event={"ID":"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f","Type":"ContainerDied","Data":"c631125ab62e34e8bf2d233d0f14d8ae6a176086b05b749fb036762ff94c4832"}
Oct 30 20:16:10.330848 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.330643    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477" event={"ID":"c3a85e15-16f9-4e1f-aa08-f1437a0bb44f","Type":"ContainerDied","Data":"09fecd3360db7ba271fb17fc6b3941fae29696f258c8829b8b38c7dfdc3a2df3"}
Oct 30 20:16:10.330848 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.330665    6030 scope.go:122] "RemoveContainer" containerID="c631125ab62e34e8bf2d233d0f14d8ae6a176086b05b749fb036762ff94c4832"
Oct 30 20:16:10.341537 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.341508    6030 scope.go:122] "RemoveContainer" containerID="c631125ab62e34e8bf2d233d0f14d8ae6a176086b05b749fb036762ff94c4832"
Oct 30 20:16:10.341850 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:16:10.341821    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c631125ab62e34e8bf2d233d0f14d8ae6a176086b05b749fb036762ff94c4832\": container with ID starting with c631125ab62e34e8bf2d233d0f14d8ae6a176086b05b749fb036762ff94c4832 not found: ID does not exist" containerID="c631125ab62e34e8bf2d233d0f14d8ae6a176086b05b749fb036762ff94c4832"
Oct 30 20:16:10.341969 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.341862    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c631125ab62e34e8bf2d233d0f14d8ae6a176086b05b749fb036762ff94c4832"} err="failed to get container status \"c631125ab62e34e8bf2d233d0f14d8ae6a176086b05b749fb036762ff94c4832\": rpc error: code = NotFound desc = could not find container \"c631125ab62e34e8bf2d233d0f14d8ae6a176086b05b749fb036762ff94c4832\": container with ID starting with c631125ab62e34e8bf2d233d0f14d8ae6a176086b05b749fb036762ff94c4832 not found: ID does not exist"
Oct 30 20:16:10.364165 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.364138    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"]
Oct 30 20:16:10.367071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:10.367033    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c8241-5dd88878c7-qt477"]
Oct 30 20:16:11.204534 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.204501    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a85e15-16f9-4e1f-aa08-f1437a0bb44f" path="/var/lib/kubelet/pods/c3a85e15-16f9-4e1f-aa08-f1437a0bb44f/volumes"
Oct 30 20:16:11.229243 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.229221    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"
Oct 30 20:16:11.293947 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.293913    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"]
Oct 30 20:16:11.295588 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.295556    6030 cpu_manager.go:414] "containerMap: removing container" podUID="c3a85e15-16f9-4e1f-aa08-f1437a0bb44f" containerName="maas-api"
Oct 30 20:16:11.295588 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.295589    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a85e15-16f9-4e1f-aa08-f1437a0bb44f" containerName="maas-api"
Oct 30 20:16:11.295812 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.295657    6030 cpu_manager.go:414] "containerMap: removing container" podUID="d4849f63-03b7-4090-bbc9-c0039ea48195" containerName="istio-proxy"
Oct 30 20:16:11.295812 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.295666    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4849f63-03b7-4090-bbc9-c0039ea48195" containerName="istio-proxy"
Oct 30 20:16:11.295812 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.295691    6030 cpu_manager.go:414] "containerMap: removing container" podUID="d4849f63-03b7-4090-bbc9-c0039ea48195" containerName="istio-proxy"
Oct 30 20:16:11.295812 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.295701    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4849f63-03b7-4090-bbc9-c0039ea48195" containerName="istio-proxy"
Oct 30 20:16:11.296747 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.296597    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a85e15-16f9-4e1f-aa08-f1437a0bb44f" containerName="maas-api"
Oct 30 20:16:11.296747 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.296630    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4849f63-03b7-4090-bbc9-c0039ea48195" containerName="istio-proxy"
Oct 30 20:16:11.296747 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.296653    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4849f63-03b7-4090-bbc9-c0039ea48195" containerName="istio-proxy"
Oct 30 20:16:11.304870 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.304848    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.319348 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.317888    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-4d2317-openshift-default-dockercfg-qqqtl\""
Oct 30 20:16:11.319348 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.318789    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"]
Oct 30 20:16:11.371080 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.370998    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istiod-ca-cert\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.371231 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.371084    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-data\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.371231 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.371195    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-podinfo\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.371231 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.371226    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-token\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.371363 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.371252    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pglvt\" (UniqueName: \"kubernetes.io/projected/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-kube-api-access-pglvt\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.371401 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.371352    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-envoy\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.371436 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.371413    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-credential-socket\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.371436 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.371430    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-workload-socket\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.371499 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.371451    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-workload-certs\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.472668 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.472591    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-podinfo\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.472668 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.472628    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-token\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.472668 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.472644    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pglvt\" (UniqueName: \"kubernetes.io/projected/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-kube-api-access-pglvt\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.472668 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.472667    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-envoy\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.472964 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.472715    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-credential-socket\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.472964 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.472732    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-workload-socket\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.472964 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.472751    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-workload-certs\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.472964 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.472940    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istiod-ca-cert\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.473236 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.472975    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-data\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.473607 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.473452    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-data\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.473607 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.473512    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-workload-socket\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.473607 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.473515    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-workload-certs\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.473607 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.473573    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-credential-socket\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.474124 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.474104    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istiod-ca-cert\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.475502 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.475474    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-podinfo\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.475584 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.475561    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-envoy\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.486828 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.486803    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-token\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.486920 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.486803    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pglvt\" (UniqueName: \"kubernetes.io/projected/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-kube-api-access-pglvt\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.633310 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.633277    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:11.923448 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.923418    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"]
Oct 30 20:16:11.923702 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:16:11.923679    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610afb5d_5aff_4fe9_bbbc_c8c82722b55d.slice/crio-4bd4b6ed28729ba2ea1898c471719a5043a64fb112fe528a2f0db4e492026799 WatchSource:0}: Error finding container 4bd4b6ed28729ba2ea1898c471719a5043a64fb112fe528a2f0db4e492026799: Status 404 returned error can't find the container with id 4bd4b6ed28729ba2ea1898c471719a5043a64fb112fe528a2f0db4e492026799
Oct 30 20:16:11.927784 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.927739    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:16:11.927864 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.927823    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:16:11.927918 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:11.927874    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:16:12.350912 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:12.350828    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m" event={"ID":"610afb5d-5aff-4fe9-bbbc-c8c82722b55d","Type":"ContainerStarted","Data":"14efaa9e6cb2697e102e7d46e40156edd0aa4339161db32a51e5fcbb4a5dda28"}
Oct 30 20:16:12.351322 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:12.350904    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m" event={"ID":"610afb5d-5aff-4fe9-bbbc-c8c82722b55d","Type":"ContainerStarted","Data":"4bd4b6ed28729ba2ea1898c471719a5043a64fb112fe528a2f0db4e492026799"}
Oct 30 20:16:12.379787 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:12.379733    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m" podStartSLOduration=1.379720165 podStartE2EDuration="1.379720165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:12.377263183 +0000 UTC m=+2651.292870039" watchObservedRunningTime="2026-10-30 20:16:12.379720165 +0000 UTC m=+2651.295327021"
Oct 30 20:16:12.633360 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:12.633333    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:12.763662 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:12.763627    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"]
Oct 30 20:16:13.635747 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:13.635716    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 20:16:13.636174 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:13.635768    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m" podUID="610afb5d-5aff-4fe9-bbbc-c8c82722b55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 20:16:14.634065 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:14.634012    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 20:16:14.634245 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:14.634100    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m" podUID="610afb5d-5aff-4fe9-bbbc-c8c82722b55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 20:16:15.633897 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:15.633858    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 20:16:15.634659 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:15.634623    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m" podUID="610afb5d-5aff-4fe9-bbbc-c8c82722b55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 20:16:16.327205 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:16.327168    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp" podUID="75ca4407-7903-475d-8c49-2dde2ba111b0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:16.634190 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:16.634155    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 20:16:16.634803 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:16.634755    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m" podUID="610afb5d-5aff-4fe9-bbbc-c8c82722b55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 20:16:17.382417 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.375396    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"]
Oct 30 20:16:17.429325 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.429295    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"]
Oct 30 20:16:17.429657 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.429636    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.441067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.433138    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.441067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.433230    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.441067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.433265    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8p2q\" (UniqueName: \"kubernetes.io/projected/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-kube-api-access-n8p2q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.441067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.433317    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.441067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.433349    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.441067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.433411    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.441067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.433454    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.454936 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.451470    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 20:16:17.454936 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.451652    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 20:16:17.454936 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.451761    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gmjw8\""
Oct 30 20:16:17.454936 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.451812    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 20:16:17.510912 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.510799    6030 generic.go:356] "Generic (PLEG): container finished" podID="64ec049e-096e-4146-8ad7-4d6a7398438f" containerID="0d9b290fbc186d166112c3928171b2c115ae258d02279a36d310caa9d4b75e88" exitCode=0
Oct 30 20:16:17.511551 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.511445    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27" event={"ID":"64ec049e-096e-4146-8ad7-4d6a7398438f","Type":"ContainerDied","Data":"0d9b290fbc186d166112c3928171b2c115ae258d02279a36d310caa9d4b75e88"}
Oct 30 20:16:17.531108 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.530963    6030 generic.go:356] "Generic (PLEG): container finished" podID="b13ed493-c629-4370-9810-7dc11c243f7f" containerID="1526ae99c146c30697ae1ef3d53feb81fd9965c28353db880b941eb6dc9cd0c4" exitCode=0
Oct 30 20:16:17.531381 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.531030    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj" event={"ID":"b13ed493-c629-4370-9810-7dc11c243f7f","Type":"ContainerDied","Data":"1526ae99c146c30697ae1ef3d53feb81fd9965c28353db880b941eb6dc9cd0c4"}
Oct 30 20:16:17.599109 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.536201    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.599109 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.536270    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.599109 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.536434    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.599109 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.536530    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.599109 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.536569    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8p2q\" (UniqueName: \"kubernetes.io/projected/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-kube-api-access-n8p2q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.599109 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.536627    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.599109 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.536660    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.599109 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.590674    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.599109 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.594456    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.640243 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.639280    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 20:16:17.640243 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.639354    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m" podUID="610afb5d-5aff-4fe9-bbbc-c8c82722b55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 20:16:17.654409 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.653223    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.655196 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.654720    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.684447 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.664869    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.684447 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.665492    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8p2q\" (UniqueName: \"kubernetes.io/projected/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-kube-api-access-n8p2q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.689315 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.689279    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9msfb\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.690946 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.690743    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:17.766994 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:17.766408    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7" podUID="f5f76d83-cf47-4e1e-a66b-a6951bbc79cf" containerName="payload-pre-processing" containerID="cri-o://0da95a002d335d790eaa42a4d1c4cd2d3c9df23084665459b255e86a8b56594e" gracePeriod=25
Oct 30 20:16:18.035744 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.035470    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg" podUID="aff9109e-c1a8-4ba0-9510-dc8d18be9547" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:18.063850 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.060207    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b" podUID="2c8b6e10-33d3-479c-b3ae-c43d2058022e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:18.238317 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.237917    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"
Oct 30 20:16:18.358559 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.354915    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"
Oct 30 20:16:18.456497 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.455667    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b13ed493-c629-4370-9810-7dc11c243f7f-kube-api-access-vnstk\" (UniqueName: \"kubernetes.io/projected/b13ed493-c629-4370-9810-7dc11c243f7f-kube-api-access-vnstk\") pod \"b13ed493-c629-4370-9810-7dc11c243f7f\" (UID: \"b13ed493-c629-4370-9810-7dc11c243f7f\") "
Oct 30 20:16:18.456497 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.455838    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b13ed493-c629-4370-9810-7dc11c243f7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13ed493-c629-4370-9810-7dc11c243f7f-plugins-config-volume\") pod \"b13ed493-c629-4370-9810-7dc11c243f7f\" (UID: \"b13ed493-c629-4370-9810-7dc11c243f7f\") "
Oct 30 20:16:18.456497 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.455909    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ec049e-096e-4146-8ad7-4d6a7398438f-kube-api-access-h4t8c\" (UniqueName: \"kubernetes.io/projected/64ec049e-096e-4146-8ad7-4d6a7398438f-kube-api-access-h4t8c\") pod \"64ec049e-096e-4146-8ad7-4d6a7398438f\" (UID: \"64ec049e-096e-4146-8ad7-4d6a7398438f\") "
Oct 30 20:16:18.456497 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.456030    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ec049e-096e-4146-8ad7-4d6a7398438f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ec049e-096e-4146-8ad7-4d6a7398438f-plugins-config-volume\") pod \"64ec049e-096e-4146-8ad7-4d6a7398438f\" (UID: \"64ec049e-096e-4146-8ad7-4d6a7398438f\") "
Oct 30 20:16:18.458922 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.457634    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ec049e-096e-4146-8ad7-4d6a7398438f-plugins-config-volume" pod "64ec049e-096e-4146-8ad7-4d6a7398438f" (UID: "64ec049e-096e-4146-8ad7-4d6a7398438f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:18.459585 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.459559    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b13ed493-c629-4370-9810-7dc11c243f7f-plugins-config-volume" pod "b13ed493-c629-4370-9810-7dc11c243f7f" (UID: "b13ed493-c629-4370-9810-7dc11c243f7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:18.466865 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.466810    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ec049e-096e-4146-8ad7-4d6a7398438f-kube-api-access-h4t8c" pod "64ec049e-096e-4146-8ad7-4d6a7398438f" (UID: "64ec049e-096e-4146-8ad7-4d6a7398438f"). InnerVolumeSpecName "kube-api-access-h4t8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:18.470929 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.470879    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b13ed493-c629-4370-9810-7dc11c243f7f-kube-api-access-vnstk" pod "b13ed493-c629-4370-9810-7dc11c243f7f" (UID: "b13ed493-c629-4370-9810-7dc11c243f7f"). InnerVolumeSpecName "kube-api-access-vnstk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:18.560123 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.559932    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13ed493-c629-4370-9810-7dc11c243f7f-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:18.560123 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.559967    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4t8c\" (UniqueName: \"kubernetes.io/projected/64ec049e-096e-4146-8ad7-4d6a7398438f-kube-api-access-h4t8c\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:18.560123 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.559991    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ec049e-096e-4146-8ad7-4d6a7398438f-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:18.560123 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.560005    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnstk\" (UniqueName: \"kubernetes.io/projected/b13ed493-c629-4370-9810-7dc11c243f7f-kube-api-access-vnstk\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:18.560123 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.560078    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27" event={"ID":"64ec049e-096e-4146-8ad7-4d6a7398438f","Type":"ContainerDied","Data":"fb44ca453e790df0a2ea23306606b608babe29fb834e22eeef6577d956ed4367"}
Oct 30 20:16:18.560546 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.560181    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"
Oct 30 20:16:18.560546 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.560198    6030 scope.go:122] "RemoveContainer" containerID="0d9b290fbc186d166112c3928171b2c115ae258d02279a36d310caa9d4b75e88"
Oct 30 20:16:18.565711 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.565653    6030 generic.go:356] "Generic (PLEG): container finished" podID="f5f76d83-cf47-4e1e-a66b-a6951bbc79cf" containerID="0da95a002d335d790eaa42a4d1c4cd2d3c9df23084665459b255e86a8b56594e" exitCode=0
Oct 30 20:16:18.566308 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.565854    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7" event={"ID":"f5f76d83-cf47-4e1e-a66b-a6951bbc79cf","Type":"ContainerDied","Data":"0da95a002d335d790eaa42a4d1c4cd2d3c9df23084665459b255e86a8b56594e"}
Oct 30 20:16:18.568771 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.568742    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj" event={"ID":"b13ed493-c629-4370-9810-7dc11c243f7f","Type":"ContainerDied","Data":"4f13e5977c39cd8b5c7d5a55dabf37f6c141077d4953185872b1d897ae5aedab"}
Oct 30 20:16:18.569060 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.568859    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"
Oct 30 20:16:18.581393 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.581368    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"]
Oct 30 20:16:18.581512 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.581466    6030 scope.go:122] "RemoveContainer" containerID="1526ae99c146c30697ae1ef3d53feb81fd9965c28353db880b941eb6dc9cd0c4"
Oct 30 20:16:18.588059 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:16:18.588020    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d2f03f_1b37_4cca_94a1_2c434d9a8f1e.slice/crio-748243a8bde0bcb70cd9e385a57a1706256f16f009f9933185401e06b7888ab1 WatchSource:0}: Error finding container 748243a8bde0bcb70cd9e385a57a1706256f16f009f9933185401e06b7888ab1: Status 404 returned error can't find the container with id 748243a8bde0bcb70cd9e385a57a1706256f16f009f9933185401e06b7888ab1
Oct 30 20:16:18.594149 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.593523    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:16:18.594149 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.593597    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:16:18.604040 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.604013    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"]
Oct 30 20:16:18.610085 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.610026    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-68685b-95c569d74-vxj27"]
Oct 30 20:16:18.618684 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.618659    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"
Oct 30 20:16:18.629592 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.629106    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"]
Oct 30 20:16:18.636695 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.636672    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-68685b-7dcc9c598c-4npjj"]
Oct 30 20:16:18.660555 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.660521    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5f76d83-cf47-4e1e-a66b-a6951bbc79cf-kube-api-access-8qttc\" (UniqueName: \"kubernetes.io/projected/f5f76d83-cf47-4e1e-a66b-a6951bbc79cf-kube-api-access-8qttc\") pod \"f5f76d83-cf47-4e1e-a66b-a6951bbc79cf\" (UID: \"f5f76d83-cf47-4e1e-a66b-a6951bbc79cf\") "
Oct 30 20:16:18.660963 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.660635    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5f76d83-cf47-4e1e-a66b-a6951bbc79cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f76d83-cf47-4e1e-a66b-a6951bbc79cf-plugins-config-volume\") pod \"f5f76d83-cf47-4e1e-a66b-a6951bbc79cf\" (UID: \"f5f76d83-cf47-4e1e-a66b-a6951bbc79cf\") "
Oct 30 20:16:18.660963 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.660950    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f76d83-cf47-4e1e-a66b-a6951bbc79cf-plugins-config-volume" pod "f5f76d83-cf47-4e1e-a66b-a6951bbc79cf" (UID: "f5f76d83-cf47-4e1e-a66b-a6951bbc79cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:18.661173 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.661156    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f76d83-cf47-4e1e-a66b-a6951bbc79cf-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:18.663551 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.663526    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f76d83-cf47-4e1e-a66b-a6951bbc79cf-kube-api-access-8qttc" pod "f5f76d83-cf47-4e1e-a66b-a6951bbc79cf" (UID: "f5f76d83-cf47-4e1e-a66b-a6951bbc79cf"). InnerVolumeSpecName "kube-api-access-8qttc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:18.738958 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.738914    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:18.739695 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.739499    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:18.740549 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.740515    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:16:18.762900 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:18.762858    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qttc\" (UniqueName: \"kubernetes.io/projected/f5f76d83-cf47-4e1e-a66b-a6951bbc79cf-kube-api-access-8qttc\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:19.203240 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:19.203169    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ec049e-096e-4146-8ad7-4d6a7398438f" path="/var/lib/kubelet/pods/64ec049e-096e-4146-8ad7-4d6a7398438f/volumes"
Oct 30 20:16:19.203638 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:19.203622    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b13ed493-c629-4370-9810-7dc11c243f7f" path="/var/lib/kubelet/pods/b13ed493-c629-4370-9810-7dc11c243f7f/volumes"
Oct 30 20:16:19.588983 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:19.588878    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb" event={"ID":"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e","Type":"ContainerStarted","Data":"bad7893287a4c88918507b1731cdea2bc8400d312a1879d3a369125426c87527"}
Oct 30 20:16:19.588983 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:19.588948    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb" event={"ID":"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e","Type":"ContainerStarted","Data":"748243a8bde0bcb70cd9e385a57a1706256f16f009f9933185401e06b7888ab1"}
Oct 30 20:16:19.591343 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:19.590235    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:19.592879 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:19.592842    6030 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9msfb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:16:19.593025 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:19.592891    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb" podUID="c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:16:19.601318 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:19.601291    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"
Oct 30 20:16:19.601446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:19.601319    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7" event={"ID":"f5f76d83-cf47-4e1e-a66b-a6951bbc79cf","Type":"ContainerDied","Data":"289b5beee9ab62b0e967fdfeec8260908b1a81a82a6accd5ea8fef7663fc582a"}
Oct 30 20:16:19.601446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:19.601376    6030 scope.go:122] "RemoveContainer" containerID="0da95a002d335d790eaa42a4d1c4cd2d3c9df23084665459b255e86a8b56594e"
Oct 30 20:16:19.626380 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:19.626318    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb" podStartSLOduration=2.626304761 podStartE2EDuration="2.626304761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:19.626280979 +0000 UTC m=+2658.541887841" watchObservedRunningTime="2026-10-30 20:16:19.626304761 +0000 UTC m=+2658.541911624"
Oct 30 20:16:19.659879 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:19.659846    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"]
Oct 30 20:16:19.664806 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:19.664305    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2a2286-68b98fcc4c-cbhc7"]
Oct 30 20:16:20.625069 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:20.619257    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:16:21.205646 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:21.205617    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5f76d83-cf47-4e1e-a66b-a6951bbc79cf" path="/var/lib/kubelet/pods/f5f76d83-cf47-4e1e-a66b-a6951bbc79cf/volumes"
Oct 30 20:16:23.641802 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:23.641771    6030 generic.go:356] "Generic (PLEG): container finished" podID="75ca4407-7903-475d-8c49-2dde2ba111b0" containerID="711d4b63f87932b08d60f4e961aa68172822d98b2ae6437f31fb22d53d4f150f" exitCode=0
Oct 30 20:16:23.642223 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:23.641847    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp" event={"ID":"75ca4407-7903-475d-8c49-2dde2ba111b0","Type":"ContainerDied","Data":"711d4b63f87932b08d60f4e961aa68172822d98b2ae6437f31fb22d53d4f150f"}
Oct 30 20:16:23.769376 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:23.769351    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"
Oct 30 20:16:23.840011 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:23.839977    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ca4407-7903-475d-8c49-2dde2ba111b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ca4407-7903-475d-8c49-2dde2ba111b0-plugins-config-volume\") pod \"75ca4407-7903-475d-8c49-2dde2ba111b0\" (UID: \"75ca4407-7903-475d-8c49-2dde2ba111b0\") "
Oct 30 20:16:23.840224 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:23.840064    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ca4407-7903-475d-8c49-2dde2ba111b0-kube-api-access-d98rz\" (UniqueName: \"kubernetes.io/projected/75ca4407-7903-475d-8c49-2dde2ba111b0-kube-api-access-d98rz\") pod \"75ca4407-7903-475d-8c49-2dde2ba111b0\" (UID: \"75ca4407-7903-475d-8c49-2dde2ba111b0\") "
Oct 30 20:16:23.840633 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:23.840589    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ca4407-7903-475d-8c49-2dde2ba111b0-plugins-config-volume" pod "75ca4407-7903-475d-8c49-2dde2ba111b0" (UID: "75ca4407-7903-475d-8c49-2dde2ba111b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:23.840857 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:23.840827    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ca4407-7903-475d-8c49-2dde2ba111b0-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:23.843161 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:23.842986    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ca4407-7903-475d-8c49-2dde2ba111b0-kube-api-access-d98rz" pod "75ca4407-7903-475d-8c49-2dde2ba111b0" (UID: "75ca4407-7903-475d-8c49-2dde2ba111b0"). InnerVolumeSpecName "kube-api-access-d98rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:23.941867 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:23.941824    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d98rz\" (UniqueName: \"kubernetes.io/projected/75ca4407-7903-475d-8c49-2dde2ba111b0-kube-api-access-d98rz\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.650484 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:24.650452    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"
Oct 30 20:16:24.651019 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:24.650488    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp" event={"ID":"75ca4407-7903-475d-8c49-2dde2ba111b0","Type":"ContainerDied","Data":"09be82df00f63d67a28edf94253c9ee10cffaf2c6474f8d3fa1817427ef99a1b"}
Oct 30 20:16:24.651019 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:24.650554    6030 scope.go:122] "RemoveContainer" containerID="711d4b63f87932b08d60f4e961aa68172822d98b2ae6437f31fb22d53d4f150f"
Oct 30 20:16:24.676769 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:24.676743    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"]
Oct 30 20:16:24.679621 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:24.679601    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c8241-c59d94c7f-gjlbp"]
Oct 30 20:16:25.203076 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:25.203021    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ca4407-7903-475d-8c49-2dde2ba111b0" path="/var/lib/kubelet/pods/75ca4407-7903-475d-8c49-2dde2ba111b0/volumes"
Oct 30 20:16:27.131020 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.130983    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"]
Oct 30 20:16:27.131560 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.131273    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp" podUID="5259686e-a598-48b6-8edc-221bcfd661e7" containerName="maas-api" containerID="cri-o://2af03603be87722f3f97373b022f3361f937d9145adf6027aff20028c54a8339" gracePeriod=30
Oct 30 20:16:27.517072 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.516244    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:16:27.597919 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.597892    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5259686e-a598-48b6-8edc-221bcfd661e7-kube-api-access-lgmtz\" (UniqueName: \"kubernetes.io/projected/5259686e-a598-48b6-8edc-221bcfd661e7-kube-api-access-lgmtz\") pod \"5259686e-a598-48b6-8edc-221bcfd661e7\" (UID: \"5259686e-a598-48b6-8edc-221bcfd661e7\") "
Oct 30 20:16:27.598130 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.597929    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5259686e-a598-48b6-8edc-221bcfd661e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5259686e-a598-48b6-8edc-221bcfd661e7-metrics-certs\") pod \"5259686e-a598-48b6-8edc-221bcfd661e7\" (UID: \"5259686e-a598-48b6-8edc-221bcfd661e7\") "
Oct 30 20:16:27.598130 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.597963    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5259686e-a598-48b6-8edc-221bcfd661e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5259686e-a598-48b6-8edc-221bcfd661e7-openshift-service-ca\") pod \"5259686e-a598-48b6-8edc-221bcfd661e7\" (UID: \"5259686e-a598-48b6-8edc-221bcfd661e7\") "
Oct 30 20:16:27.598130 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.598031    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5259686e-a598-48b6-8edc-221bcfd661e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5259686e-a598-48b6-8edc-221bcfd661e7-maas-api-tls\") pod \"5259686e-a598-48b6-8edc-221bcfd661e7\" (UID: \"5259686e-a598-48b6-8edc-221bcfd661e7\") "
Oct 30 20:16:27.598445 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.598421    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5259686e-a598-48b6-8edc-221bcfd661e7-openshift-service-ca" pod "5259686e-a598-48b6-8edc-221bcfd661e7" (UID: "5259686e-a598-48b6-8edc-221bcfd661e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:27.598789 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.598703    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5259686e-a598-48b6-8edc-221bcfd661e7-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:27.600571 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.600520    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5259686e-a598-48b6-8edc-221bcfd661e7-maas-api-tls" pod "5259686e-a598-48b6-8edc-221bcfd661e7" (UID: "5259686e-a598-48b6-8edc-221bcfd661e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:27.600654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.600634    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5259686e-a598-48b6-8edc-221bcfd661e7-kube-api-access-lgmtz" pod "5259686e-a598-48b6-8edc-221bcfd661e7" (UID: "5259686e-a598-48b6-8edc-221bcfd661e7"). InnerVolumeSpecName "kube-api-access-lgmtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:27.600654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.600631    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5259686e-a598-48b6-8edc-221bcfd661e7-metrics-certs" pod "5259686e-a598-48b6-8edc-221bcfd661e7" (UID: "5259686e-a598-48b6-8edc-221bcfd661e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:27.674154 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.674076    6030 generic.go:356] "Generic (PLEG): container finished" podID="5259686e-a598-48b6-8edc-221bcfd661e7" containerID="2af03603be87722f3f97373b022f3361f937d9145adf6027aff20028c54a8339" exitCode=0
Oct 30 20:16:27.674306 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.674201    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"
Oct 30 20:16:27.674306 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.674221    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp" event={"ID":"5259686e-a598-48b6-8edc-221bcfd661e7","Type":"ContainerDied","Data":"2af03603be87722f3f97373b022f3361f937d9145adf6027aff20028c54a8339"}
Oct 30 20:16:27.674306 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.674289    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp" event={"ID":"5259686e-a598-48b6-8edc-221bcfd661e7","Type":"ContainerDied","Data":"41eacdd9220299ae05f01943ea5d8e465f526eef29abd02c1aa4f4a9d23b1257"}
Oct 30 20:16:27.674447 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.674321    6030 scope.go:122] "RemoveContainer" containerID="2af03603be87722f3f97373b022f3361f937d9145adf6027aff20028c54a8339"
Oct 30 20:16:27.686108 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.686089    6030 scope.go:122] "RemoveContainer" containerID="2af03603be87722f3f97373b022f3361f937d9145adf6027aff20028c54a8339"
Oct 30 20:16:27.686396 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:16:27.686378    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af03603be87722f3f97373b022f3361f937d9145adf6027aff20028c54a8339\": container with ID starting with 2af03603be87722f3f97373b022f3361f937d9145adf6027aff20028c54a8339 not found: ID does not exist" containerID="2af03603be87722f3f97373b022f3361f937d9145adf6027aff20028c54a8339"
Oct 30 20:16:27.686503 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.686408    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af03603be87722f3f97373b022f3361f937d9145adf6027aff20028c54a8339"} err="failed to get container status \"2af03603be87722f3f97373b022f3361f937d9145adf6027aff20028c54a8339\": rpc error: code = NotFound desc = could not find container \"2af03603be87722f3f97373b022f3361f937d9145adf6027aff20028c54a8339\": container with ID starting with 2af03603be87722f3f97373b022f3361f937d9145adf6027aff20028c54a8339 not found: ID does not exist"
Oct 30 20:16:27.700083 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.700014    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5259686e-a598-48b6-8edc-221bcfd661e7-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:27.700083 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.700068    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgmtz\" (UniqueName: \"kubernetes.io/projected/5259686e-a598-48b6-8edc-221bcfd661e7-kube-api-access-lgmtz\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:27.700083 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.700083    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5259686e-a598-48b6-8edc-221bcfd661e7-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:27.707035 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.707005    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"]
Oct 30 20:16:27.713424 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:27.713397    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2a2286-79d7b65c45-x28dp"]
Oct 30 20:16:28.011334 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:28.011228    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b" podUID="2c8b6e10-33d3-479c-b3ae-c43d2058022e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:28.011518 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:28.011380    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg" podUID="aff9109e-c1a8-4ba0-9510-dc8d18be9547" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:29.202873 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:29.202835    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5259686e-a598-48b6-8edc-221bcfd661e7" path="/var/lib/kubelet/pods/5259686e-a598-48b6-8edc-221bcfd661e7/volumes"
Oct 30 20:16:29.739434 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:16:29.739396    6030 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff9109e_c1a8_4ba0_9510_dc8d18be9547.slice/crio-conmon-a4e877eed84af948eba7b99ae2c76f8651a42ef07b86664a116c3dd80cca05fc.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:16:29.905426 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:29.905392    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"
Oct 30 20:16:29.975335 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:29.973262    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"
Oct 30 20:16:30.030393 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.030360    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff9109e-c1a8-4ba0-9510-dc8d18be9547-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff9109e-c1a8-4ba0-9510-dc8d18be9547-plugins-config-volume\") pod \"aff9109e-c1a8-4ba0-9510-dc8d18be9547\" (UID: \"aff9109e-c1a8-4ba0-9510-dc8d18be9547\") "
Oct 30 20:16:30.030644 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.030621    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff9109e-c1a8-4ba0-9510-dc8d18be9547-kube-api-access-jvxsw\" (UniqueName: \"kubernetes.io/projected/aff9109e-c1a8-4ba0-9510-dc8d18be9547-kube-api-access-jvxsw\") pod \"aff9109e-c1a8-4ba0-9510-dc8d18be9547\" (UID: \"aff9109e-c1a8-4ba0-9510-dc8d18be9547\") "
Oct 30 20:16:30.030807 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.030781    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff9109e-c1a8-4ba0-9510-dc8d18be9547-plugins-config-volume" pod "aff9109e-c1a8-4ba0-9510-dc8d18be9547" (UID: "aff9109e-c1a8-4ba0-9510-dc8d18be9547"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:30.031332 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.031293    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff9109e-c1a8-4ba0-9510-dc8d18be9547-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:30.033918 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.033890    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff9109e-c1a8-4ba0-9510-dc8d18be9547-kube-api-access-jvxsw" pod "aff9109e-c1a8-4ba0-9510-dc8d18be9547" (UID: "aff9109e-c1a8-4ba0-9510-dc8d18be9547"). InnerVolumeSpecName "kube-api-access-jvxsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:30.132277 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.132193    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8b6e10-33d3-479c-b3ae-c43d2058022e-kube-api-access-p4kd7\" (UniqueName: \"kubernetes.io/projected/2c8b6e10-33d3-479c-b3ae-c43d2058022e-kube-api-access-p4kd7\") pod \"2c8b6e10-33d3-479c-b3ae-c43d2058022e\" (UID: \"2c8b6e10-33d3-479c-b3ae-c43d2058022e\") "
Oct 30 20:16:30.132436 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.132329    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8b6e10-33d3-479c-b3ae-c43d2058022e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8b6e10-33d3-479c-b3ae-c43d2058022e-plugins-config-volume\") pod \"2c8b6e10-33d3-479c-b3ae-c43d2058022e\" (UID: \"2c8b6e10-33d3-479c-b3ae-c43d2058022e\") "
Oct 30 20:16:30.132668 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.132643    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8b6e10-33d3-479c-b3ae-c43d2058022e-plugins-config-volume" pod "2c8b6e10-33d3-479c-b3ae-c43d2058022e" (UID: "2c8b6e10-33d3-479c-b3ae-c43d2058022e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:30.132919 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.132898    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvxsw\" (UniqueName: \"kubernetes.io/projected/aff9109e-c1a8-4ba0-9510-dc8d18be9547-kube-api-access-jvxsw\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:30.132985 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.132924    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8b6e10-33d3-479c-b3ae-c43d2058022e-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:30.134817 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.134789    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8b6e10-33d3-479c-b3ae-c43d2058022e-kube-api-access-p4kd7" pod "2c8b6e10-33d3-479c-b3ae-c43d2058022e" (UID: "2c8b6e10-33d3-479c-b3ae-c43d2058022e"). InnerVolumeSpecName "kube-api-access-p4kd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:30.234305 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.234268    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4kd7\" (UniqueName: \"kubernetes.io/projected/2c8b6e10-33d3-479c-b3ae-c43d2058022e-kube-api-access-p4kd7\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:30.724090 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.724030    6030 generic.go:356] "Generic (PLEG): container finished" podID="aff9109e-c1a8-4ba0-9510-dc8d18be9547" containerID="a4e877eed84af948eba7b99ae2c76f8651a42ef07b86664a116c3dd80cca05fc" exitCode=0
Oct 30 20:16:30.724268 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.724129    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"
Oct 30 20:16:30.724268 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.724149    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg" event={"ID":"aff9109e-c1a8-4ba0-9510-dc8d18be9547","Type":"ContainerDied","Data":"a4e877eed84af948eba7b99ae2c76f8651a42ef07b86664a116c3dd80cca05fc"}
Oct 30 20:16:30.724268 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.724218    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg" event={"ID":"aff9109e-c1a8-4ba0-9510-dc8d18be9547","Type":"ContainerDied","Data":"7297574a63896c26b000caddce33100770b9f4ede59577bc1be38407fae6f763"}
Oct 30 20:16:30.724268 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.724259    6030 scope.go:122] "RemoveContainer" containerID="a4e877eed84af948eba7b99ae2c76f8651a42ef07b86664a116c3dd80cca05fc"
Oct 30 20:16:30.726264 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.726242    6030 generic.go:356] "Generic (PLEG): container finished" podID="2c8b6e10-33d3-479c-b3ae-c43d2058022e" containerID="33369a845a4b39dea21c8a43bc226c714fefebd69f16a210ed69dff2af6379f0" exitCode=0
Oct 30 20:16:30.726394 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.726292    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b" event={"ID":"2c8b6e10-33d3-479c-b3ae-c43d2058022e","Type":"ContainerDied","Data":"33369a845a4b39dea21c8a43bc226c714fefebd69f16a210ed69dff2af6379f0"}
Oct 30 20:16:30.726394 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.726334    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b" event={"ID":"2c8b6e10-33d3-479c-b3ae-c43d2058022e","Type":"ContainerDied","Data":"c295563fd5ff7a0124821e950c68b823950c4191f80dd8811cd32aaedf429ec2"}
Oct 30 20:16:30.726522 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.726396    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"
Oct 30 20:16:30.749824 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.749777    6030 scope.go:122] "RemoveContainer" containerID="a4e877eed84af948eba7b99ae2c76f8651a42ef07b86664a116c3dd80cca05fc"
Oct 30 20:16:30.751255 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:16:30.751219    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e877eed84af948eba7b99ae2c76f8651a42ef07b86664a116c3dd80cca05fc\": container with ID starting with a4e877eed84af948eba7b99ae2c76f8651a42ef07b86664a116c3dd80cca05fc not found: ID does not exist" containerID="a4e877eed84af948eba7b99ae2c76f8651a42ef07b86664a116c3dd80cca05fc"
Oct 30 20:16:30.751394 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.751262    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e877eed84af948eba7b99ae2c76f8651a42ef07b86664a116c3dd80cca05fc"} err="failed to get container status \"a4e877eed84af948eba7b99ae2c76f8651a42ef07b86664a116c3dd80cca05fc\": rpc error: code = NotFound desc = could not find container \"a4e877eed84af948eba7b99ae2c76f8651a42ef07b86664a116c3dd80cca05fc\": container with ID starting with a4e877eed84af948eba7b99ae2c76f8651a42ef07b86664a116c3dd80cca05fc not found: ID does not exist"
Oct 30 20:16:30.751394 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.751287    6030 scope.go:122] "RemoveContainer" containerID="33369a845a4b39dea21c8a43bc226c714fefebd69f16a210ed69dff2af6379f0"
Oct 30 20:16:30.758463 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.758430    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"]
Oct 30 20:16:30.764454 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.764429    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c9bce8-6b8df985dd-75fgg"]
Oct 30 20:16:30.767913 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.767886    6030 scope.go:122] "RemoveContainer" containerID="33369a845a4b39dea21c8a43bc226c714fefebd69f16a210ed69dff2af6379f0"
Oct 30 20:16:30.768325 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:16:30.768296    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33369a845a4b39dea21c8a43bc226c714fefebd69f16a210ed69dff2af6379f0\": container with ID starting with 33369a845a4b39dea21c8a43bc226c714fefebd69f16a210ed69dff2af6379f0 not found: ID does not exist" containerID="33369a845a4b39dea21c8a43bc226c714fefebd69f16a210ed69dff2af6379f0"
Oct 30 20:16:30.768432 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.768330    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33369a845a4b39dea21c8a43bc226c714fefebd69f16a210ed69dff2af6379f0"} err="failed to get container status \"33369a845a4b39dea21c8a43bc226c714fefebd69f16a210ed69dff2af6379f0\": rpc error: code = NotFound desc = could not find container \"33369a845a4b39dea21c8a43bc226c714fefebd69f16a210ed69dff2af6379f0\": container with ID starting with 33369a845a4b39dea21c8a43bc226c714fefebd69f16a210ed69dff2af6379f0 not found: ID does not exist"
Oct 30 20:16:30.777637 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.777589    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"]
Oct 30 20:16:30.782576 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:30.782550    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c9bce8-6c76d4788c-9845b"]
Oct 30 20:16:31.204266 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:31.204235    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8b6e10-33d3-479c-b3ae-c43d2058022e" path="/var/lib/kubelet/pods/2c8b6e10-33d3-479c-b3ae-c43d2058022e/volumes"
Oct 30 20:16:31.204643 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:31.204629    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff9109e-c1a8-4ba0-9510-dc8d18be9547" path="/var/lib/kubelet/pods/aff9109e-c1a8-4ba0-9510-dc8d18be9547/volumes"
Oct 30 20:16:32.024935 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:32.024891    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"]
Oct 30 20:16:32.053670 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:32.053640    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"]
Oct 30 20:16:34.291792 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:16:34.291751    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:16:35.772762 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:35.772728    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"]
Oct 30 20:16:35.773293 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:35.772938    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk" podUID="cdeef59a-a6c5-49c9-a081-8d374f01166d" containerName="istio-proxy" containerID="cri-o://be7d269465d17b685543a6f5b5b4e0667c15cc99a0429dc276ce7e4f6881c5bb" gracePeriod=30
Oct 30 20:16:37.026061 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:37.025997    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb" podUID="3fbd9f0c-7c56-406e-8dd9-0770f49f4878" containerName="payload-processing" containerID="cri-o://6f6f089747e4170d17f004e1521b44faa435d2ea587ffe7cf8a0d883664f3caf" gracePeriod=25
Oct 30 20:16:37.054225 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:37.054166    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm" podUID="0118f805-4deb-4de0-aaf2-cad5ba1ed2fa" containerName="payload-pre-processing" containerID="cri-o://802b482b1cb150e52fc52450a4f7fbf82e86e9ddd6b3416c89e120f9e7401101" gracePeriod=25
Oct 30 20:16:37.998652 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:37.998614    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm" podUID="0118f805-4deb-4de0-aaf2-cad5ba1ed2fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:39.008353 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:39.008308    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb" podUID="3fbd9f0c-7c56-406e-8dd9-0770f49f4878" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:41.002704 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.002659    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"]
Oct 30 20:16:41.088329 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.088300    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:16:41.203653 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.203621    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdeef59a-a6c5-49c9-a081-8d374f01166d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdeef59a-a6c5-49c9-a081-8d374f01166d-istiod-ca-cert\") pod \"cdeef59a-a6c5-49c9-a081-8d374f01166d\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") "
Oct 30 20:16:41.203832 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.203713    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdeef59a-a6c5-49c9-a081-8d374f01166d-kube-api-access-44t7d\" (UniqueName: \"kubernetes.io/projected/cdeef59a-a6c5-49c9-a081-8d374f01166d-kube-api-access-44t7d\") pod \"cdeef59a-a6c5-49c9-a081-8d374f01166d\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") "
Oct 30 20:16:41.203832 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.203757    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-credential-socket\") pod \"cdeef59a-a6c5-49c9-a081-8d374f01166d\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") "
Oct 30 20:16:41.203945 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.203832    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-data\") pod \"cdeef59a-a6c5-49c9-a081-8d374f01166d\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") "
Oct 30 20:16:41.203945 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.203871    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-envoy\") pod \"cdeef59a-a6c5-49c9-a081-8d374f01166d\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") "
Oct 30 20:16:41.203945 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.203900    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-token\" (UniqueName: \"kubernetes.io/projected/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-token\") pod \"cdeef59a-a6c5-49c9-a081-8d374f01166d\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") "
Oct 30 20:16:41.204082 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.203955    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-workload-socket\") pod \"cdeef59a-a6c5-49c9-a081-8d374f01166d\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") "
Oct 30 20:16:41.204082 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.203980    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-podinfo\") pod \"cdeef59a-a6c5-49c9-a081-8d374f01166d\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") "
Oct 30 20:16:41.204082 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.203990    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdeef59a-a6c5-49c9-a081-8d374f01166d-istiod-ca-cert" pod "cdeef59a-a6c5-49c9-a081-8d374f01166d" (UID: "cdeef59a-a6c5-49c9-a081-8d374f01166d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:41.204082 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.204017    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-workload-certs\") pod \"cdeef59a-a6c5-49c9-a081-8d374f01166d\" (UID: \"cdeef59a-a6c5-49c9-a081-8d374f01166d\") "
Oct 30 20:16:41.204454 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.204403    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-workload-socket" pod "cdeef59a-a6c5-49c9-a081-8d374f01166d" (UID: "cdeef59a-a6c5-49c9-a081-8d374f01166d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:41.204454 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.204421    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-data" pod "cdeef59a-a6c5-49c9-a081-8d374f01166d" (UID: "cdeef59a-a6c5-49c9-a081-8d374f01166d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:41.204694 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.204575    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:41.204694 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.204596    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:41.204694 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.204663    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdeef59a-a6c5-49c9-a081-8d374f01166d-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:41.204873 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.204770    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-workload-certs" pod "cdeef59a-a6c5-49c9-a081-8d374f01166d" (UID: "cdeef59a-a6c5-49c9-a081-8d374f01166d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:41.204873 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.204864    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-credential-socket" pod "cdeef59a-a6c5-49c9-a081-8d374f01166d" (UID: "cdeef59a-a6c5-49c9-a081-8d374f01166d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:41.206506 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.206480    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-podinfo" pod "cdeef59a-a6c5-49c9-a081-8d374f01166d" (UID: "cdeef59a-a6c5-49c9-a081-8d374f01166d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:16:41.206739 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.206718    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-envoy" pod "cdeef59a-a6c5-49c9-a081-8d374f01166d" (UID: "cdeef59a-a6c5-49c9-a081-8d374f01166d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:41.206922 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.206902    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdeef59a-a6c5-49c9-a081-8d374f01166d-kube-api-access-44t7d" pod "cdeef59a-a6c5-49c9-a081-8d374f01166d" (UID: "cdeef59a-a6c5-49c9-a081-8d374f01166d"). InnerVolumeSpecName "kube-api-access-44t7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:41.207121 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.207106    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-token" pod "cdeef59a-a6c5-49c9-a081-8d374f01166d" (UID: "cdeef59a-a6c5-49c9-a081-8d374f01166d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:41.308127 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.307026    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:41.308127 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.307085    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:41.308127 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.307100    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdeef59a-a6c5-49c9-a081-8d374f01166d-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:41.308127 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.307115    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:41.308127 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.307130    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44t7d\" (UniqueName: \"kubernetes.io/projected/cdeef59a-a6c5-49c9-a081-8d374f01166d-kube-api-access-44t7d\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:41.308127 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.307144    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeef59a-a6c5-49c9-a081-8d374f01166d-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:41.811068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.811027    6030 generic.go:356] "Generic (PLEG): container finished" podID="cdeef59a-a6c5-49c9-a081-8d374f01166d" containerID="be7d269465d17b685543a6f5b5b4e0667c15cc99a0429dc276ce7e4f6881c5bb" exitCode=0
Oct 30 20:16:41.811245 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.811089    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"
Oct 30 20:16:41.811245 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.811107    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk" event={"ID":"cdeef59a-a6c5-49c9-a081-8d374f01166d","Type":"ContainerDied","Data":"be7d269465d17b685543a6f5b5b4e0667c15cc99a0429dc276ce7e4f6881c5bb"}
Oct 30 20:16:41.811245 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.811162    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk" event={"ID":"cdeef59a-a6c5-49c9-a081-8d374f01166d","Type":"ContainerDied","Data":"476cba67a4e0ef2606a07f2259f3f98e756dabba06396f9828c84cebaa813520"}
Oct 30 20:16:41.811245 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.811186    6030 scope.go:122] "RemoveContainer" containerID="be7d269465d17b685543a6f5b5b4e0667c15cc99a0429dc276ce7e4f6881c5bb"
Oct 30 20:16:41.822761 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.822742    6030 scope.go:122] "RemoveContainer" containerID="be7d269465d17b685543a6f5b5b4e0667c15cc99a0429dc276ce7e4f6881c5bb"
Oct 30 20:16:41.823016 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:16:41.822997    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be7d269465d17b685543a6f5b5b4e0667c15cc99a0429dc276ce7e4f6881c5bb\": container with ID starting with be7d269465d17b685543a6f5b5b4e0667c15cc99a0429dc276ce7e4f6881c5bb not found: ID does not exist" containerID="be7d269465d17b685543a6f5b5b4e0667c15cc99a0429dc276ce7e4f6881c5bb"
Oct 30 20:16:41.823080 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.823026    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be7d269465d17b685543a6f5b5b4e0667c15cc99a0429dc276ce7e4f6881c5bb"} err="failed to get container status \"be7d269465d17b685543a6f5b5b4e0667c15cc99a0429dc276ce7e4f6881c5bb\": rpc error: code = NotFound desc = could not find container \"be7d269465d17b685543a6f5b5b4e0667c15cc99a0429dc276ce7e4f6881c5bb\": container with ID starting with be7d269465d17b685543a6f5b5b4e0667c15cc99a0429dc276ce7e4f6881c5bb not found: ID does not exist"
Oct 30 20:16:41.838375 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.838352    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"]
Oct 30 20:16:41.843319 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:41.843299    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-pcdqk"]
Oct 30 20:16:43.203176 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:43.203145    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdeef59a-a6c5-49c9-a081-8d374f01166d" path="/var/lib/kubelet/pods/cdeef59a-a6c5-49c9-a081-8d374f01166d/volumes"
Oct 30 20:16:45.400071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:45.399433    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"]
Oct 30 20:16:45.400071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:45.399662    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8" podUID="b259c906-057d-47dd-ace7-630c8ade49fb" containerName="sim" containerID="cri-o://f223640f6ecceca05121631dc89de2372712d3d75366ea09431c1ad79a186a48" gracePeriod=30
Oct 30 20:16:45.861559 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:45.861534    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"
Oct 30 20:16:45.927846 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:45.927813    6030 generic.go:356] "Generic (PLEG): container finished" podID="b259c906-057d-47dd-ace7-630c8ade49fb" containerID="f223640f6ecceca05121631dc89de2372712d3d75366ea09431c1ad79a186a48" exitCode=0
Oct 30 20:16:45.927846 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:45.927851    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"
Oct 30 20:16:45.928091 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:45.927896    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8" event={"ID":"b259c906-057d-47dd-ace7-630c8ade49fb","Type":"ContainerDied","Data":"f223640f6ecceca05121631dc89de2372712d3d75366ea09431c1ad79a186a48"}
Oct 30 20:16:45.928091 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:45.927935    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8" event={"ID":"b259c906-057d-47dd-ace7-630c8ade49fb","Type":"ContainerDied","Data":"cfa757a153acd71443c00f589ae046635ac2260ebc90aba289482677117362c9"}
Oct 30 20:16:45.928091 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:45.927957    6030 scope.go:122] "RemoveContainer" containerID="f223640f6ecceca05121631dc89de2372712d3d75366ea09431c1ad79a186a48"
Oct 30 20:16:45.940171 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:45.940147    6030 scope.go:122] "RemoveContainer" containerID="f223640f6ecceca05121631dc89de2372712d3d75366ea09431c1ad79a186a48"
Oct 30 20:16:45.940462 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:16:45.940440    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f223640f6ecceca05121631dc89de2372712d3d75366ea09431c1ad79a186a48\": container with ID starting with f223640f6ecceca05121631dc89de2372712d3d75366ea09431c1ad79a186a48 not found: ID does not exist" containerID="f223640f6ecceca05121631dc89de2372712d3d75366ea09431c1ad79a186a48"
Oct 30 20:16:45.940535 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:45.940473    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f223640f6ecceca05121631dc89de2372712d3d75366ea09431c1ad79a186a48"} err="failed to get container status \"f223640f6ecceca05121631dc89de2372712d3d75366ea09431c1ad79a186a48\": rpc error: code = NotFound desc = could not find container \"f223640f6ecceca05121631dc89de2372712d3d75366ea09431c1ad79a186a48\": container with ID starting with f223640f6ecceca05121631dc89de2372712d3d75366ea09431c1ad79a186a48 not found: ID does not exist"
Oct 30 20:16:46.012496 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:46.003216    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc" podUID="0b80bf01-a7f4-41ae-9adc-f550c6ec7707" containerName="payload-processing" containerID="cri-o://f73e85ae5b581eb4e54401dbdcb77d11b8667b94823158ccd159fd0d2062cf0a" gracePeriod=25
Oct 30 20:16:46.015803 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:46.014034    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b259c906-057d-47dd-ace7-630c8ade49fb-kube-api-access-br7gv\" (UniqueName: \"kubernetes.io/projected/b259c906-057d-47dd-ace7-630c8ade49fb-kube-api-access-br7gv\") pod \"b259c906-057d-47dd-ace7-630c8ade49fb\" (UID: \"b259c906-057d-47dd-ace7-630c8ade49fb\") "
Oct 30 20:16:46.015803 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:46.014140    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b259c906-057d-47dd-ace7-630c8ade49fb-tls\" (UniqueName: \"kubernetes.io/secret/b259c906-057d-47dd-ace7-630c8ade49fb-tls\") pod \"b259c906-057d-47dd-ace7-630c8ade49fb\" (UID: \"b259c906-057d-47dd-ace7-630c8ade49fb\") "
Oct 30 20:16:46.017393 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:46.017364    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b259c906-057d-47dd-ace7-630c8ade49fb-kube-api-access-br7gv" pod "b259c906-057d-47dd-ace7-630c8ade49fb" (UID: "b259c906-057d-47dd-ace7-630c8ade49fb"). InnerVolumeSpecName "kube-api-access-br7gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:46.020399 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:46.020373    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b259c906-057d-47dd-ace7-630c8ade49fb-tls" pod "b259c906-057d-47dd-ace7-630c8ade49fb" (UID: "b259c906-057d-47dd-ace7-630c8ade49fb"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:46.116024 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:46.115978    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br7gv\" (UniqueName: \"kubernetes.io/projected/b259c906-057d-47dd-ace7-630c8ade49fb-kube-api-access-br7gv\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:46.116024 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:46.116020    6030 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b259c906-057d-47dd-ace7-630c8ade49fb-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:46.259423 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:46.259385    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"]
Oct 30 20:16:46.266632 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:46.266574    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-a41ade/e2e-usage-sim-1b1eae-774689b76c-fljn8"]
Oct 30 20:16:47.220073 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:47.216746    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b259c906-057d-47dd-ace7-630c8ade49fb" path="/var/lib/kubelet/pods/b259c906-057d-47dd-ace7-630c8ade49fb/volumes"
Oct 30 20:16:47.998476 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:47.998440    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm" podUID="0118f805-4deb-4de0-aaf2-cad5ba1ed2fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:49.008365 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:49.008322    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb" podUID="3fbd9f0c-7c56-406e-8dd9-0770f49f4878" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:49.519848 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:49.519813    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc" podUID="0b80bf01-a7f4-41ae-9adc-f550c6ec7707" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:52.328090 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.326994    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"]
Oct 30 20:16:52.328090 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.327304    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz" podUID="ef621cd9-9b74-457a-a134-472f7a7b89ca" containerName="maas-api" containerID="cri-o://07fa8a6c13ff702f7ab83ab1bc3122590c69b8964ea24b9cc44fd79b16474143" gracePeriod=30
Oct 30 20:16:52.751088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.750511    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:16:52.793450 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.793410    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef621cd9-9b74-457a-a134-472f7a7b89ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef621cd9-9b74-457a-a134-472f7a7b89ca-maas-api-tls\") pod \"ef621cd9-9b74-457a-a134-472f7a7b89ca\" (UID: \"ef621cd9-9b74-457a-a134-472f7a7b89ca\") "
Oct 30 20:16:52.793658 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.793630    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef621cd9-9b74-457a-a134-472f7a7b89ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef621cd9-9b74-457a-a134-472f7a7b89ca-openshift-service-ca\") pod \"ef621cd9-9b74-457a-a134-472f7a7b89ca\" (UID: \"ef621cd9-9b74-457a-a134-472f7a7b89ca\") "
Oct 30 20:16:52.793730 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.793706    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef621cd9-9b74-457a-a134-472f7a7b89ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef621cd9-9b74-457a-a134-472f7a7b89ca-metrics-certs\") pod \"ef621cd9-9b74-457a-a134-472f7a7b89ca\" (UID: \"ef621cd9-9b74-457a-a134-472f7a7b89ca\") "
Oct 30 20:16:52.793812 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.793794    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef621cd9-9b74-457a-a134-472f7a7b89ca-kube-api-access-bbhnf\" (UniqueName: \"kubernetes.io/projected/ef621cd9-9b74-457a-a134-472f7a7b89ca-kube-api-access-bbhnf\") pod \"ef621cd9-9b74-457a-a134-472f7a7b89ca\" (UID: \"ef621cd9-9b74-457a-a134-472f7a7b89ca\") "
Oct 30 20:16:52.794534 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.794488    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef621cd9-9b74-457a-a134-472f7a7b89ca-openshift-service-ca" pod "ef621cd9-9b74-457a-a134-472f7a7b89ca" (UID: "ef621cd9-9b74-457a-a134-472f7a7b89ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:52.796367 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.796344    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef621cd9-9b74-457a-a134-472f7a7b89ca-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:52.798311 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.798282    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef621cd9-9b74-457a-a134-472f7a7b89ca-maas-api-tls" pod "ef621cd9-9b74-457a-a134-472f7a7b89ca" (UID: "ef621cd9-9b74-457a-a134-472f7a7b89ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:52.798420 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.798402    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef621cd9-9b74-457a-a134-472f7a7b89ca-kube-api-access-bbhnf" pod "ef621cd9-9b74-457a-a134-472f7a7b89ca" (UID: "ef621cd9-9b74-457a-a134-472f7a7b89ca"). InnerVolumeSpecName "kube-api-access-bbhnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:52.798504 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.798485    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef621cd9-9b74-457a-a134-472f7a7b89ca-metrics-certs" pod "ef621cd9-9b74-457a-a134-472f7a7b89ca" (UID: "ef621cd9-9b74-457a-a134-472f7a7b89ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:52.897756 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.897719    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbhnf\" (UniqueName: \"kubernetes.io/projected/ef621cd9-9b74-457a-a134-472f7a7b89ca-kube-api-access-bbhnf\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:52.897756 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.897750    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef621cd9-9b74-457a-a134-472f7a7b89ca-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:52.897756 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.897760    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef621cd9-9b74-457a-a134-472f7a7b89ca-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:52.986480 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.986389    6030 generic.go:356] "Generic (PLEG): container finished" podID="ef621cd9-9b74-457a-a134-472f7a7b89ca" containerID="07fa8a6c13ff702f7ab83ab1bc3122590c69b8964ea24b9cc44fd79b16474143" exitCode=0
Oct 30 20:16:52.986480 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.986452    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"
Oct 30 20:16:52.986480 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.986473    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz" event={"ID":"ef621cd9-9b74-457a-a134-472f7a7b89ca","Type":"ContainerDied","Data":"07fa8a6c13ff702f7ab83ab1bc3122590c69b8964ea24b9cc44fd79b16474143"}
Oct 30 20:16:52.986778 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.986531    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz" event={"ID":"ef621cd9-9b74-457a-a134-472f7a7b89ca","Type":"ContainerDied","Data":"164fb3a976bf5cf75f8856bbb95c696bcc3cd9ed9cccf2e8510f409fc705892e"}
Oct 30 20:16:52.986778 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.986557    6030 scope.go:122] "RemoveContainer" containerID="07fa8a6c13ff702f7ab83ab1bc3122590c69b8964ea24b9cc44fd79b16474143"
Oct 30 20:16:52.997934 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.997917    6030 scope.go:122] "RemoveContainer" containerID="07fa8a6c13ff702f7ab83ab1bc3122590c69b8964ea24b9cc44fd79b16474143"
Oct 30 20:16:52.998281 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:16:52.998253    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fa8a6c13ff702f7ab83ab1bc3122590c69b8964ea24b9cc44fd79b16474143\": container with ID starting with 07fa8a6c13ff702f7ab83ab1bc3122590c69b8964ea24b9cc44fd79b16474143 not found: ID does not exist" containerID="07fa8a6c13ff702f7ab83ab1bc3122590c69b8964ea24b9cc44fd79b16474143"
Oct 30 20:16:52.998368 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:52.998283    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fa8a6c13ff702f7ab83ab1bc3122590c69b8964ea24b9cc44fd79b16474143"} err="failed to get container status \"07fa8a6c13ff702f7ab83ab1bc3122590c69b8964ea24b9cc44fd79b16474143\": rpc error: code = NotFound desc = could not find container \"07fa8a6c13ff702f7ab83ab1bc3122590c69b8964ea24b9cc44fd79b16474143\": container with ID starting with 07fa8a6c13ff702f7ab83ab1bc3122590c69b8964ea24b9cc44fd79b16474143 not found: ID does not exist"
Oct 30 20:16:53.013161 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:53.013129    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"]
Oct 30 20:16:53.017829 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:53.017807    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d97cc8-5b6df5565b-27ljz"]
Oct 30 20:16:53.205212 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:53.205183    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef621cd9-9b74-457a-a134-472f7a7b89ca" path="/var/lib/kubelet/pods/ef621cd9-9b74-457a-a134-472f7a7b89ca/volumes"
Oct 30 20:16:55.004161 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.004130    6030 generic.go:356] "Generic (PLEG): container finished" podID="6484295e-f0bf-4194-9416-12c15129dd4a" containerID="9c46eb3acfb2f2da869e5cd6f8096318a029748e3ee24ea055bcdeb0cb05cccf" exitCode=0
Oct 30 20:16:55.004661 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.004200    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2" event={"ID":"6484295e-f0bf-4194-9416-12c15129dd4a","Type":"ContainerDied","Data":"9c46eb3acfb2f2da869e5cd6f8096318a029748e3ee24ea055bcdeb0cb05cccf"}
Oct 30 20:16:55.004661 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.004624    6030 scope.go:122] "RemoveContainer" containerID="9c46eb3acfb2f2da869e5cd6f8096318a029748e3ee24ea055bcdeb0cb05cccf"
Oct 30 20:16:55.006465 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.006430    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:16:55.006569 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.006505    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:16:55.006569 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.006538    6030 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"}
Oct 30 20:16:55.263484 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.263399    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"]
Oct 30 20:16:55.264391 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264367    6030 cpu_manager.go:414] "containerMap: removing container" podUID="64ec049e-096e-4146-8ad7-4d6a7398438f" containerName="payload-processing"
Oct 30 20:16:55.264391 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264389    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ec049e-096e-4146-8ad7-4d6a7398438f" containerName="payload-processing"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264407    6030 cpu_manager.go:414] "containerMap: removing container" podUID="b259c906-057d-47dd-ace7-630c8ade49fb" containerName="sim"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264414    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b259c906-057d-47dd-ace7-630c8ade49fb" containerName="sim"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264433    6030 cpu_manager.go:414] "containerMap: removing container" podUID="2c8b6e10-33d3-479c-b3ae-c43d2058022e" containerName="payload-pre-processing"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264441    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8b6e10-33d3-479c-b3ae-c43d2058022e" containerName="payload-pre-processing"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264455    6030 cpu_manager.go:414] "containerMap: removing container" podUID="5259686e-a598-48b6-8edc-221bcfd661e7" containerName="maas-api"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264464    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5259686e-a598-48b6-8edc-221bcfd661e7" containerName="maas-api"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264485    6030 cpu_manager.go:414] "containerMap: removing container" podUID="75ca4407-7903-475d-8c49-2dde2ba111b0" containerName="payload-pre-processing"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264493    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ca4407-7903-475d-8c49-2dde2ba111b0" containerName="payload-pre-processing"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264521    6030 cpu_manager.go:414] "containerMap: removing container" podUID="f5f76d83-cf47-4e1e-a66b-a6951bbc79cf" containerName="payload-pre-processing"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264528    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5f76d83-cf47-4e1e-a66b-a6951bbc79cf" containerName="payload-pre-processing"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264554    6030 cpu_manager.go:414] "containerMap: removing container" podUID="cdeef59a-a6c5-49c9-a081-8d374f01166d" containerName="istio-proxy"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264560    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdeef59a-a6c5-49c9-a081-8d374f01166d" containerName="istio-proxy"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264572    6030 cpu_manager.go:414] "containerMap: removing container" podUID="ef621cd9-9b74-457a-a134-472f7a7b89ca" containerName="maas-api"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264578    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef621cd9-9b74-457a-a134-472f7a7b89ca" containerName="maas-api"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264606    6030 cpu_manager.go:414] "containerMap: removing container" podUID="b13ed493-c629-4370-9810-7dc11c243f7f" containerName="payload-pre-processing"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264613    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13ed493-c629-4370-9810-7dc11c243f7f" containerName="payload-pre-processing"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264625    6030 cpu_manager.go:414] "containerMap: removing container" podUID="aff9109e-c1a8-4ba0-9510-dc8d18be9547" containerName="payload-processing"
Oct 30 20:16:55.264623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264632    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff9109e-c1a8-4ba0-9510-dc8d18be9547" containerName="payload-processing"
Oct 30 20:16:55.265550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264764    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ec049e-096e-4146-8ad7-4d6a7398438f" containerName="payload-processing"
Oct 30 20:16:55.265550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264778    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="b259c906-057d-47dd-ace7-630c8ade49fb" containerName="sim"
Oct 30 20:16:55.265550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264790    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8b6e10-33d3-479c-b3ae-c43d2058022e" containerName="payload-pre-processing"
Oct 30 20:16:55.265550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264799    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="5259686e-a598-48b6-8edc-221bcfd661e7" containerName="maas-api"
Oct 30 20:16:55.265550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264811    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ca4407-7903-475d-8c49-2dde2ba111b0" containerName="payload-pre-processing"
Oct 30 20:16:55.265550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264822    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5f76d83-cf47-4e1e-a66b-a6951bbc79cf" containerName="payload-pre-processing"
Oct 30 20:16:55.265550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264833    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdeef59a-a6c5-49c9-a081-8d374f01166d" containerName="istio-proxy"
Oct 30 20:16:55.265550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264841    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef621cd9-9b74-457a-a134-472f7a7b89ca" containerName="maas-api"
Oct 30 20:16:55.265550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264851    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="b13ed493-c629-4370-9810-7dc11c243f7f" containerName="payload-pre-processing"
Oct 30 20:16:55.265550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.264862    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff9109e-c1a8-4ba0-9510-dc8d18be9547" containerName="payload-processing"
Oct 30 20:16:55.269326 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.269303    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.279099 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.277862    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f63d20\"/\"default-dockercfg-mp5wd\""
Oct 30 20:16:55.282482 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.282251    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"]
Oct 30 20:16:55.282482 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.282315    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f63d20\"/\"test-model-f63d20-kserve-self-signed-certs\""
Oct 30 20:16:55.282734 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.282714    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f63d20\"/\"openshift-service-ca.crt\""
Oct 30 20:16:55.282854 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.282826    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f63d20\"/\"kube-root-ca.crt\""
Oct 30 20:16:55.335163 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.335091    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-model-cache\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.335367 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.335215    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-tmp-dir\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.335367 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.335268    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-dshm\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.335480 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.335426    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-home\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.335574 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.335554    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkl62\" (UniqueName: \"kubernetes.io/projected/f9310b6e-0c69-4e69-89a9-a268f132aa69-kube-api-access-lkl62\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.335775 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.335755    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9310b6e-0c69-4e69-89a9-a268f132aa69-tls-certs\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.439237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.438035    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkl62\" (UniqueName: \"kubernetes.io/projected/f9310b6e-0c69-4e69-89a9-a268f132aa69-kube-api-access-lkl62\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.439237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.438356    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9310b6e-0c69-4e69-89a9-a268f132aa69-tls-certs\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.439237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.438457    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-model-cache\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.439237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.438548    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-tmp-dir\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.439237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.438590    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-dshm\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.439237 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.438624    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-home\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.439710 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.439250    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-home\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.447076 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.443398    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-tmp-dir\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.447076 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.443666    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-model-cache\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.475093 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.471989    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-dshm\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.499202 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.494750    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9310b6e-0c69-4e69-89a9-a268f132aa69-tls-certs\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.499202 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.495390    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkl62\" (UniqueName: \"kubernetes.io/projected/f9310b6e-0c69-4e69-89a9-a268f132aa69-kube-api-access-lkl62\") pod \"test-model-f63d20-kserve-84bdf84565-276kj\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") " pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.597168 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.597032    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:55.914248 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:55.914219    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"]
Oct 30 20:16:55.915891 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:16:55.915862    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9310b6e_0c69_4e69_89a9_a268f132aa69.slice/crio-96c8a8a0704a4ad32ae360d96a41d7634ac29276d24e13dc136ad36db96e6951 WatchSource:0}: Error finding container 96c8a8a0704a4ad32ae360d96a41d7634ac29276d24e13dc136ad36db96e6951: Status 404 returned error can't find the container with id 96c8a8a0704a4ad32ae360d96a41d7634ac29276d24e13dc136ad36db96e6951
Oct 30 20:16:56.026074 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:56.025486    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2" event={"ID":"6484295e-f0bf-4194-9416-12c15129dd4a","Type":"ContainerStarted","Data":"c87e8156656784a350cdde7d73ae6b4d785111a89281b1152d43aa55f67ad99c"}
Oct 30 20:16:56.031404 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:56.031376    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj" event={"ID":"f9310b6e-0c69-4e69-89a9-a268f132aa69","Type":"ContainerStarted","Data":"96c8a8a0704a4ad32ae360d96a41d7634ac29276d24e13dc136ad36db96e6951"}
Oct 30 20:16:56.088221 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:56.088152    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj" podStartSLOduration=1.088130646 podStartE2EDuration="1.088130646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:56.077437281 +0000 UTC m=+2694.993044138" watchObservedRunningTime="2026-10-30 20:16:56.088130646 +0000 UTC m=+2695.003737515"
Oct 30 20:16:56.597416 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:56.597376    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:16:56.683069 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:56.680396    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:16:56.684068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:56.683730    6030 patch_prober.go:29] interesting pod/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 20:16:56.684068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:56.683787    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2" podUID="6484295e-f0bf-4194-9416-12c15129dd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 20:16:57.048591 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.048560    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj" event={"ID":"f9310b6e-0c69-4e69-89a9-a268f132aa69","Type":"ContainerStarted","Data":"fff00d669592485b50e4cf407b18033a58cd16fbfb9b99838919fbcd973590cd"}
Oct 30 20:16:57.196667 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.196193    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:16:57.444594 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.444574    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"
Oct 30 20:16:57.460143 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.460119    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"
Oct 30 20:16:57.600340 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.600211    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fbd9f0c-7c56-406e-8dd9-0770f49f4878-kube-api-access-xjf8s\" (UniqueName: \"kubernetes.io/projected/3fbd9f0c-7c56-406e-8dd9-0770f49f4878-kube-api-access-xjf8s\") pod \"3fbd9f0c-7c56-406e-8dd9-0770f49f4878\" (UID: \"3fbd9f0c-7c56-406e-8dd9-0770f49f4878\") "
Oct 30 20:16:57.600840 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.600817    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0118f805-4deb-4de0-aaf2-cad5ba1ed2fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0118f805-4deb-4de0-aaf2-cad5ba1ed2fa-plugins-config-volume\") pod \"0118f805-4deb-4de0-aaf2-cad5ba1ed2fa\" (UID: \"0118f805-4deb-4de0-aaf2-cad5ba1ed2fa\") "
Oct 30 20:16:57.601107 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.601088    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fbd9f0c-7c56-406e-8dd9-0770f49f4878-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fbd9f0c-7c56-406e-8dd9-0770f49f4878-plugins-config-volume\") pod \"3fbd9f0c-7c56-406e-8dd9-0770f49f4878\" (UID: \"3fbd9f0c-7c56-406e-8dd9-0770f49f4878\") "
Oct 30 20:16:57.601554 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.601511    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0118f805-4deb-4de0-aaf2-cad5ba1ed2fa-kube-api-access-pb7mt\" (UniqueName: \"kubernetes.io/projected/0118f805-4deb-4de0-aaf2-cad5ba1ed2fa-kube-api-access-pb7mt\") pod \"0118f805-4deb-4de0-aaf2-cad5ba1ed2fa\" (UID: \"0118f805-4deb-4de0-aaf2-cad5ba1ed2fa\") "
Oct 30 20:16:57.601897 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.601588    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbd9f0c-7c56-406e-8dd9-0770f49f4878-plugins-config-volume" pod "3fbd9f0c-7c56-406e-8dd9-0770f49f4878" (UID: "3fbd9f0c-7c56-406e-8dd9-0770f49f4878"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:57.601897 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.601774    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0118f805-4deb-4de0-aaf2-cad5ba1ed2fa-plugins-config-volume" pod "0118f805-4deb-4de0-aaf2-cad5ba1ed2fa" (UID: "0118f805-4deb-4de0-aaf2-cad5ba1ed2fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:57.602509 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.602437    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0118f805-4deb-4de0-aaf2-cad5ba1ed2fa-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:57.602610 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.602517    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fbd9f0c-7c56-406e-8dd9-0770f49f4878-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:57.604290 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.604263    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0118f805-4deb-4de0-aaf2-cad5ba1ed2fa-kube-api-access-pb7mt" pod "0118f805-4deb-4de0-aaf2-cad5ba1ed2fa" (UID: "0118f805-4deb-4de0-aaf2-cad5ba1ed2fa"). InnerVolumeSpecName "kube-api-access-pb7mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:57.604399 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.604371    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbd9f0c-7c56-406e-8dd9-0770f49f4878-kube-api-access-xjf8s" pod "3fbd9f0c-7c56-406e-8dd9-0770f49f4878" (UID: "3fbd9f0c-7c56-406e-8dd9-0770f49f4878"). InnerVolumeSpecName "kube-api-access-xjf8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:57.697144 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.694029    6030 patch_prober.go:29] interesting pod/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 20:16:57.697332 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.697196    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2" podUID="6484295e-f0bf-4194-9416-12c15129dd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 20:16:57.704061 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.704016    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb7mt\" (UniqueName: \"kubernetes.io/projected/0118f805-4deb-4de0-aaf2-cad5ba1ed2fa-kube-api-access-pb7mt\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:57.704204 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:57.704070    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjf8s\" (UniqueName: \"kubernetes.io/projected/3fbd9f0c-7c56-406e-8dd9-0770f49f4878-kube-api-access-xjf8s\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:58.057611 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.057579    6030 generic.go:356] "Generic (PLEG): container finished" podID="0118f805-4deb-4de0-aaf2-cad5ba1ed2fa" containerID="802b482b1cb150e52fc52450a4f7fbf82e86e9ddd6b3416c89e120f9e7401101" exitCode=0
Oct 30 20:16:58.058121 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.057627    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"
Oct 30 20:16:58.058121 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.057654    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm" event={"ID":"0118f805-4deb-4de0-aaf2-cad5ba1ed2fa","Type":"ContainerDied","Data":"802b482b1cb150e52fc52450a4f7fbf82e86e9ddd6b3416c89e120f9e7401101"}
Oct 30 20:16:58.058121 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.057719    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm" event={"ID":"0118f805-4deb-4de0-aaf2-cad5ba1ed2fa","Type":"ContainerDied","Data":"bbb6ab8b85645d9271f6eed81cd3cc63b8813a84c40e97a529b151e984ca3be1"}
Oct 30 20:16:58.058121 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.057753    6030 scope.go:122] "RemoveContainer" containerID="802b482b1cb150e52fc52450a4f7fbf82e86e9ddd6b3416c89e120f9e7401101"
Oct 30 20:16:58.059338 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.059315    6030 generic.go:356] "Generic (PLEG): container finished" podID="3fbd9f0c-7c56-406e-8dd9-0770f49f4878" containerID="6f6f089747e4170d17f004e1521b44faa435d2ea587ffe7cf8a0d883664f3caf" exitCode=0
Oct 30 20:16:58.059513 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.059371    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"
Oct 30 20:16:58.059640 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.059388    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb" event={"ID":"3fbd9f0c-7c56-406e-8dd9-0770f49f4878","Type":"ContainerDied","Data":"6f6f089747e4170d17f004e1521b44faa435d2ea587ffe7cf8a0d883664f3caf"}
Oct 30 20:16:58.059640 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.059627    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb" event={"ID":"3fbd9f0c-7c56-406e-8dd9-0770f49f4878","Type":"ContainerDied","Data":"f539596b3906e0dba5908e1a3828d1a3f77adc8d1ea710829f6856b376bf5a11"}
Oct 30 20:16:58.087227 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.087201    6030 scope.go:122] "RemoveContainer" containerID="802b482b1cb150e52fc52450a4f7fbf82e86e9ddd6b3416c89e120f9e7401101"
Oct 30 20:16:58.088062 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:16:58.088015    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"802b482b1cb150e52fc52450a4f7fbf82e86e9ddd6b3416c89e120f9e7401101\": container with ID starting with 802b482b1cb150e52fc52450a4f7fbf82e86e9ddd6b3416c89e120f9e7401101 not found: ID does not exist" containerID="802b482b1cb150e52fc52450a4f7fbf82e86e9ddd6b3416c89e120f9e7401101"
Oct 30 20:16:58.088892 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.088861    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"802b482b1cb150e52fc52450a4f7fbf82e86e9ddd6b3416c89e120f9e7401101"} err="failed to get container status \"802b482b1cb150e52fc52450a4f7fbf82e86e9ddd6b3416c89e120f9e7401101\": rpc error: code = NotFound desc = could not find container \"802b482b1cb150e52fc52450a4f7fbf82e86e9ddd6b3416c89e120f9e7401101\": container with ID starting with 802b482b1cb150e52fc52450a4f7fbf82e86e9ddd6b3416c89e120f9e7401101 not found: ID does not exist"
Oct 30 20:16:58.089035 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.089021    6030 scope.go:122] "RemoveContainer" containerID="6f6f089747e4170d17f004e1521b44faa435d2ea587ffe7cf8a0d883664f3caf"
Oct 30 20:16:58.105110 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.105086    6030 scope.go:122] "RemoveContainer" containerID="6f6f089747e4170d17f004e1521b44faa435d2ea587ffe7cf8a0d883664f3caf"
Oct 30 20:16:58.105459 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:16:58.105422    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f6f089747e4170d17f004e1521b44faa435d2ea587ffe7cf8a0d883664f3caf\": container with ID starting with 6f6f089747e4170d17f004e1521b44faa435d2ea587ffe7cf8a0d883664f3caf not found: ID does not exist" containerID="6f6f089747e4170d17f004e1521b44faa435d2ea587ffe7cf8a0d883664f3caf"
Oct 30 20:16:58.105550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.105471    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f6f089747e4170d17f004e1521b44faa435d2ea587ffe7cf8a0d883664f3caf"} err="failed to get container status \"6f6f089747e4170d17f004e1521b44faa435d2ea587ffe7cf8a0d883664f3caf\": rpc error: code = NotFound desc = could not find container \"6f6f089747e4170d17f004e1521b44faa435d2ea587ffe7cf8a0d883664f3caf\": container with ID starting with 6f6f089747e4170d17f004e1521b44faa435d2ea587ffe7cf8a0d883664f3caf not found: ID does not exist"
Oct 30 20:16:58.116201 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.116178    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"]
Oct 30 20:16:58.121553 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.121520    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-969878-686cb65c47-6gjpm"]
Oct 30 20:16:58.134605 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.134582    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"]
Oct 30 20:16:58.142828 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.142806    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-969878-5fbf8d98cb-l6tkb"]
Oct 30 20:16:58.688071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.687418    6030 patch_prober.go:29] interesting pod/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 20:16:58.688071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:58.687493    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2" podUID="6484295e-f0bf-4194-9416-12c15129dd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 20:16:59.204400 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:59.204361    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0118f805-4deb-4de0-aaf2-cad5ba1ed2fa" path="/var/lib/kubelet/pods/0118f805-4deb-4de0-aaf2-cad5ba1ed2fa/volumes"
Oct 30 20:16:59.204866 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:59.204847    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fbd9f0c-7c56-406e-8dd9-0770f49f4878" path="/var/lib/kubelet/pods/3fbd9f0c-7c56-406e-8dd9-0770f49f4878/volumes"
Oct 30 20:16:59.520363 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:59.520274    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc" podUID="0b80bf01-a7f4-41ae-9adc-f550c6ec7707" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:59.688070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:59.684391    6030 patch_prober.go:29] interesting pod/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 20:16:59.688070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:16:59.684451    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2" podUID="6484295e-f0bf-4194-9416-12c15129dd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 20:17:00.696068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:00.689165    6030 patch_prober.go:29] interesting pod/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 20:17:00.696068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:00.689228    6030 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2" podUID="6484295e-f0bf-4194-9416-12c15129dd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 20:17:01.686624 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:01.686584    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:17:01.688446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:01.688417    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:17:02.143353 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:02.143322    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"]
Oct 30 20:17:02.163308 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:02.163278    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"]
Oct 30 20:17:04.290546 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:17:04.290513    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:17:05.598122 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:05.598089    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:17:05.610644 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:05.610621    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:17:06.177984 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:06.177581    6030 generic.go:356] "Generic (PLEG): container finished" podID="0b80bf01-a7f4-41ae-9adc-f550c6ec7707" containerID="f73e85ae5b581eb4e54401dbdcb77d11b8667b94823158ccd159fd0d2062cf0a" exitCode=0
Oct 30 20:17:06.178189 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:06.178135    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc" event={"ID":"0b80bf01-a7f4-41ae-9adc-f550c6ec7707","Type":"ContainerDied","Data":"f73e85ae5b581eb4e54401dbdcb77d11b8667b94823158ccd159fd0d2062cf0a"}
Oct 30 20:17:06.193130 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:06.192290    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:17:06.278654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:06.278575    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"
Oct 30 20:17:06.474098 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:06.450674    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b80bf01-a7f4-41ae-9adc-f550c6ec7707-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b80bf01-a7f4-41ae-9adc-f550c6ec7707-plugins-config-volume\") pod \"0b80bf01-a7f4-41ae-9adc-f550c6ec7707\" (UID: \"0b80bf01-a7f4-41ae-9adc-f550c6ec7707\") "
Oct 30 20:17:06.474098 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:06.450861    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b80bf01-a7f4-41ae-9adc-f550c6ec7707-kube-api-access-nv8l7\" (UniqueName: \"kubernetes.io/projected/0b80bf01-a7f4-41ae-9adc-f550c6ec7707-kube-api-access-nv8l7\") pod \"0b80bf01-a7f4-41ae-9adc-f550c6ec7707\" (UID: \"0b80bf01-a7f4-41ae-9adc-f550c6ec7707\") "
Oct 30 20:17:06.490573 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:06.478323    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b80bf01-a7f4-41ae-9adc-f550c6ec7707-kube-api-access-nv8l7" pod "0b80bf01-a7f4-41ae-9adc-f550c6ec7707" (UID: "0b80bf01-a7f4-41ae-9adc-f550c6ec7707"). InnerVolumeSpecName "kube-api-access-nv8l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:06.490573 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:06.478828    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b80bf01-a7f4-41ae-9adc-f550c6ec7707-plugins-config-volume" pod "0b80bf01-a7f4-41ae-9adc-f550c6ec7707" (UID: "0b80bf01-a7f4-41ae-9adc-f550c6ec7707"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:06.553325 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:06.553239    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b80bf01-a7f4-41ae-9adc-f550c6ec7707-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:06.553325 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:06.553267    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv8l7\" (UniqueName: \"kubernetes.io/projected/0b80bf01-a7f4-41ae-9adc-f550c6ec7707-kube-api-access-nv8l7\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:07.144148 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:07.144084    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv" podUID="bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35" containerName="payload-processing" containerID="cri-o://7520eb36410c2ce41608bc2e4b0265e0958af5dfbec9bdc40a01097b1c76f0f1" gracePeriod=25
Oct 30 20:17:07.164516 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:07.164461    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs" podUID="20147f52-0076-4eb0-aad7-5f262111a38b" containerName="payload-pre-processing" containerID="cri-o://e0c09b9c4034a92b473c1cc58e3c81d3352ac5ae3e5d3345b0a38670161170f5" gracePeriod=25
Oct 30 20:17:07.187596 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:07.187564    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc" event={"ID":"0b80bf01-a7f4-41ae-9adc-f550c6ec7707","Type":"ContainerDied","Data":"b283e5499f2aefa577fdf11ef06e294eef9b38bcf720f847a10403297f380238"}
Oct 30 20:17:07.187596 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:07.187583    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"
Oct 30 20:17:07.187793 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:07.187665    6030 scope.go:122] "RemoveContainer" containerID="f73e85ae5b581eb4e54401dbdcb77d11b8667b94823158ccd159fd0d2062cf0a"
Oct 30 20:17:07.215477 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:07.215453    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"]
Oct 30 20:17:07.222096 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:07.222072    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d97cc8-758b89656b-k6rxc"]
Oct 30 20:17:09.204793 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:09.204761    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b80bf01-a7f4-41ae-9adc-f550c6ec7707" path="/var/lib/kubelet/pods/0b80bf01-a7f4-41ae-9adc-f550c6ec7707/volumes"
Oct 30 20:17:11.921420 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:11.921380    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"]
Oct 30 20:17:12.652144 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:12.652101    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv" podUID="bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:17:12.652352 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:12.652165    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs" podUID="20147f52-0076-4eb0-aad7-5f262111a38b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:17:15.285524 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:15.285483    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"]
Oct 30 20:17:15.286103 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:15.285749    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw" podUID="8790b73a-43cc-496c-ab9e-250aabb2306f" containerName="maas-api" containerID="cri-o://856abb2d76bf642b2bb6e582652bfb2295068e91e7c1d1aa95ab46be5fdda1da" gracePeriod=30
Oct 30 20:17:15.684068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:15.683991    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 20:17:15.792300 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:15.792263    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8790b73a-43cc-496c-ab9e-250aabb2306f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8790b73a-43cc-496c-ab9e-250aabb2306f-maas-api-tls\") pod \"8790b73a-43cc-496c-ab9e-250aabb2306f\" (UID: \"8790b73a-43cc-496c-ab9e-250aabb2306f\") "
Oct 30 20:17:15.792504 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:15.792486    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8790b73a-43cc-496c-ab9e-250aabb2306f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8790b73a-43cc-496c-ab9e-250aabb2306f-metrics-certs\") pod \"8790b73a-43cc-496c-ab9e-250aabb2306f\" (UID: \"8790b73a-43cc-496c-ab9e-250aabb2306f\") "
Oct 30 20:17:15.792641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:15.792625    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8790b73a-43cc-496c-ab9e-250aabb2306f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8790b73a-43cc-496c-ab9e-250aabb2306f-openshift-service-ca\") pod \"8790b73a-43cc-496c-ab9e-250aabb2306f\" (UID: \"8790b73a-43cc-496c-ab9e-250aabb2306f\") "
Oct 30 20:17:15.792722 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:15.792667    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8790b73a-43cc-496c-ab9e-250aabb2306f-kube-api-access-7sxhf\" (UniqueName: \"kubernetes.io/projected/8790b73a-43cc-496c-ab9e-250aabb2306f-kube-api-access-7sxhf\") pod \"8790b73a-43cc-496c-ab9e-250aabb2306f\" (UID: \"8790b73a-43cc-496c-ab9e-250aabb2306f\") "
Oct 30 20:17:15.792974 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:15.792942    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8790b73a-43cc-496c-ab9e-250aabb2306f-openshift-service-ca" pod "8790b73a-43cc-496c-ab9e-250aabb2306f" (UID: "8790b73a-43cc-496c-ab9e-250aabb2306f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:15.793455 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:15.793271    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8790b73a-43cc-496c-ab9e-250aabb2306f-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:15.794961 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:15.794931    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8790b73a-43cc-496c-ab9e-250aabb2306f-metrics-certs" pod "8790b73a-43cc-496c-ab9e-250aabb2306f" (UID: "8790b73a-43cc-496c-ab9e-250aabb2306f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:17:15.795121 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:15.795075    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8790b73a-43cc-496c-ab9e-250aabb2306f-maas-api-tls" pod "8790b73a-43cc-496c-ab9e-250aabb2306f" (UID: "8790b73a-43cc-496c-ab9e-250aabb2306f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:17:15.795478 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:15.795442    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8790b73a-43cc-496c-ab9e-250aabb2306f-kube-api-access-7sxhf" pod "8790b73a-43cc-496c-ab9e-250aabb2306f" (UID: "8790b73a-43cc-496c-ab9e-250aabb2306f"). InnerVolumeSpecName "kube-api-access-7sxhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:15.894723 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:15.894682    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8790b73a-43cc-496c-ab9e-250aabb2306f-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:15.894723 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:15.894721    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sxhf\" (UniqueName: \"kubernetes.io/projected/8790b73a-43cc-496c-ab9e-250aabb2306f-kube-api-access-7sxhf\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:15.894723 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:15.894732    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8790b73a-43cc-496c-ab9e-250aabb2306f-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:16.331672 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:16.331583    6030 generic.go:356] "Generic (PLEG): container finished" podID="8790b73a-43cc-496c-ab9e-250aabb2306f" containerID="856abb2d76bf642b2bb6e582652bfb2295068e91e7c1d1aa95ab46be5fdda1da" exitCode=0
Oct 30 20:17:16.331672 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:16.331639    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"
Oct 30 20:17:16.331672 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:16.331648    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw" event={"ID":"8790b73a-43cc-496c-ab9e-250aabb2306f","Type":"ContainerDied","Data":"856abb2d76bf642b2bb6e582652bfb2295068e91e7c1d1aa95ab46be5fdda1da"}
Oct 30 20:17:16.332205 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:16.331701    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw" event={"ID":"8790b73a-43cc-496c-ab9e-250aabb2306f","Type":"ContainerDied","Data":"b8b0f68d7e5e902ffea081eb9964ffe00c7344cb0308f8f2f90f37192c29a122"}
Oct 30 20:17:16.332205 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:16.331722    6030 scope.go:122] "RemoveContainer" containerID="856abb2d76bf642b2bb6e582652bfb2295068e91e7c1d1aa95ab46be5fdda1da"
Oct 30 20:17:16.343272 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:16.343252    6030 scope.go:122] "RemoveContainer" containerID="856abb2d76bf642b2bb6e582652bfb2295068e91e7c1d1aa95ab46be5fdda1da"
Oct 30 20:17:16.343545 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:17:16.343517    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856abb2d76bf642b2bb6e582652bfb2295068e91e7c1d1aa95ab46be5fdda1da\": container with ID starting with 856abb2d76bf642b2bb6e582652bfb2295068e91e7c1d1aa95ab46be5fdda1da not found: ID does not exist" containerID="856abb2d76bf642b2bb6e582652bfb2295068e91e7c1d1aa95ab46be5fdda1da"
Oct 30 20:17:16.343646 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:16.343561    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856abb2d76bf642b2bb6e582652bfb2295068e91e7c1d1aa95ab46be5fdda1da"} err="failed to get container status \"856abb2d76bf642b2bb6e582652bfb2295068e91e7c1d1aa95ab46be5fdda1da\": rpc error: code = NotFound desc = could not find container \"856abb2d76bf642b2bb6e582652bfb2295068e91e7c1d1aa95ab46be5fdda1da\": container with ID starting with 856abb2d76bf642b2bb6e582652bfb2295068e91e7c1d1aa95ab46be5fdda1da not found: ID does not exist"
Oct 30 20:17:16.363239 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:16.363209    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"]
Oct 30 20:17:16.365803 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:16.365782    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0eeba5-b8684d4bb-njzjw"]
Oct 30 20:17:16.921973 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:16.921931    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz" podUID="aea211f6-4038-4d8f-a09f-581a68b6c73d" containerName="payload-pre-processing" containerID="cri-o://6836db3b82359fac828050257f1f58f7df30fc63fec5d50c8f1d7c46383c25d4" gracePeriod=25
Oct 30 20:17:17.192800 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:17.192738    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"
Oct 30 20:17:17.202250 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:17.202225    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8790b73a-43cc-496c-ab9e-250aabb2306f" path="/var/lib/kubelet/pods/8790b73a-43cc-496c-ab9e-250aabb2306f/volumes"
Oct 30 20:17:17.315262 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:17.315227    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea211f6-4038-4d8f-a09f-581a68b6c73d-kube-api-access-wgj8v\" (UniqueName: \"kubernetes.io/projected/aea211f6-4038-4d8f-a09f-581a68b6c73d-kube-api-access-wgj8v\") pod \"aea211f6-4038-4d8f-a09f-581a68b6c73d\" (UID: \"aea211f6-4038-4d8f-a09f-581a68b6c73d\") "
Oct 30 20:17:17.315476 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:17.315293    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aea211f6-4038-4d8f-a09f-581a68b6c73d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea211f6-4038-4d8f-a09f-581a68b6c73d-plugins-config-volume\") pod \"aea211f6-4038-4d8f-a09f-581a68b6c73d\" (UID: \"aea211f6-4038-4d8f-a09f-581a68b6c73d\") "
Oct 30 20:17:17.315707 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:17.315674    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea211f6-4038-4d8f-a09f-581a68b6c73d-plugins-config-volume" pod "aea211f6-4038-4d8f-a09f-581a68b6c73d" (UID: "aea211f6-4038-4d8f-a09f-581a68b6c73d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:17.317585 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:17.317555    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea211f6-4038-4d8f-a09f-581a68b6c73d-kube-api-access-wgj8v" pod "aea211f6-4038-4d8f-a09f-581a68b6c73d" (UID: "aea211f6-4038-4d8f-a09f-581a68b6c73d"). InnerVolumeSpecName "kube-api-access-wgj8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:17.340405 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:17.340381    6030 generic.go:356] "Generic (PLEG): container finished" podID="aea211f6-4038-4d8f-a09f-581a68b6c73d" containerID="6836db3b82359fac828050257f1f58f7df30fc63fec5d50c8f1d7c46383c25d4" exitCode=0
Oct 30 20:17:17.340790 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:17.340429    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"
Oct 30 20:17:17.340790 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:17.340464    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz" event={"ID":"aea211f6-4038-4d8f-a09f-581a68b6c73d","Type":"ContainerDied","Data":"6836db3b82359fac828050257f1f58f7df30fc63fec5d50c8f1d7c46383c25d4"}
Oct 30 20:17:17.340790 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:17.340535    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz" event={"ID":"aea211f6-4038-4d8f-a09f-581a68b6c73d","Type":"ContainerDied","Data":"7a0acb08ac9944f20cc703e6a9a1dcf50f5cd687bf8e1f3dea4d9d4a9f45ac71"}
Oct 30 20:17:17.340790 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:17.340569    6030 scope.go:122] "RemoveContainer" containerID="6836db3b82359fac828050257f1f58f7df30fc63fec5d50c8f1d7c46383c25d4"
Oct 30 20:17:17.350367 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:17.350351    6030 scope.go:122] "RemoveContainer" containerID="6836db3b82359fac828050257f1f58f7df30fc63fec5d50c8f1d7c46383c25d4"
Oct 30 20:17:17.350640 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:17:17.350620    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6836db3b82359fac828050257f1f58f7df30fc63fec5d50c8f1d7c46383c25d4\": container with ID starting with 6836db3b82359fac828050257f1f58f7df30fc63fec5d50c8f1d7c46383c25d4 not found: ID does not exist" containerID="6836db3b82359fac828050257f1f58f7df30fc63fec5d50c8f1d7c46383c25d4"
Oct 30 20:17:17.350682 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:17.350647    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6836db3b82359fac828050257f1f58f7df30fc63fec5d50c8f1d7c46383c25d4"} err="failed to get container status \"6836db3b82359fac828050257f1f58f7df30fc63fec5d50c8f1d7c46383c25d4\": rpc error: code = NotFound desc = could not find container \"6836db3b82359fac828050257f1f58f7df30fc63fec5d50c8f1d7c46383c25d4\": container with ID starting with 6836db3b82359fac828050257f1f58f7df30fc63fec5d50c8f1d7c46383c25d4 not found: ID does not exist"
Oct 30 20:17:17.369070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:17.369018    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"]
Oct 30 20:17:17.371758 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:17.371734    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0eeba5-f7fdd5595-9xplz"]
Oct 30 20:17:17.417214 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:17.417186    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea211f6-4038-4d8f-a09f-581a68b6c73d-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:17.417214 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:17.417210    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgj8v\" (UniqueName: \"kubernetes.io/projected/aea211f6-4038-4d8f-a09f-581a68b6c73d-kube-api-access-wgj8v\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:19.202338 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:19.202304    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aea211f6-4038-4d8f-a09f-581a68b6c73d" path="/var/lib/kubelet/pods/aea211f6-4038-4d8f-a09f-581a68b6c73d/volumes"
Oct 30 20:17:22.651592 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:22.651544    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv" podUID="bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:17:22.651592 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:22.651573    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs" podUID="20147f52-0076-4eb0-aad7-5f262111a38b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:17:25.012076 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:25.007602    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"]
Oct 30 20:17:25.012076 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:25.007860    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm" podUID="c75bdfb0-89bd-4935-baae-55475fdb405a" containerName="istio-proxy" containerID="cri-o://be0e816709503de473df06f1d40d141a1ce92f3db035d05f848040644c047e18" gracePeriod=30
Oct 30 20:17:25.055141 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:25.055106    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"]
Oct 30 20:17:25.064350 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:25.064320    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"]
Oct 30 20:17:25.181384 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:25.181320    6030 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:17:25.181576 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:25.181398    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm" podUID="c75bdfb0-89bd-4935-baae-55475fdb405a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:17:27.418850 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.418717    6030 generic.go:356] "Generic (PLEG): container finished" podID="bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35" containerID="7520eb36410c2ce41608bc2e4b0265e0958af5dfbec9bdc40a01097b1c76f0f1" exitCode=0
Oct 30 20:17:27.418850 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.418801    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv" event={"ID":"bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35","Type":"ContainerDied","Data":"7520eb36410c2ce41608bc2e4b0265e0958af5dfbec9bdc40a01097b1c76f0f1"}
Oct 30 20:17:27.421096 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.421072    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs" event={"ID":"20147f52-0076-4eb0-aad7-5f262111a38b","Type":"ContainerDied","Data":"e0c09b9c4034a92b473c1cc58e3c81d3352ac5ae3e5d3345b0a38670161170f5"}
Oct 30 20:17:27.421214 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.421032    6030 generic.go:356] "Generic (PLEG): container finished" podID="20147f52-0076-4eb0-aad7-5f262111a38b" containerID="e0c09b9c4034a92b473c1cc58e3c81d3352ac5ae3e5d3345b0a38670161170f5" exitCode=0
Oct 30 20:17:27.439535 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.439516    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"
Oct 30 20:17:27.459576 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.459554    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35-kube-api-access-zrtfb\" (UniqueName: \"kubernetes.io/projected/bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35-kube-api-access-zrtfb\") pod \"bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35\" (UID: \"bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35\") "
Oct 30 20:17:27.459672 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.459659    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35-plugins-config-volume\") pod \"bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35\" (UID: \"bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35\") "
Oct 30 20:17:27.460077 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.460034    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35-plugins-config-volume" pod "bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35" (UID: "bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:27.461256 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.461231    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:27.462364 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.462338    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35-kube-api-access-zrtfb" pod "bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35" (UID: "bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35"). InnerVolumeSpecName "kube-api-access-zrtfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:27.473903 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.473884    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"
Oct 30 20:17:27.569647 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.569612    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20147f52-0076-4eb0-aad7-5f262111a38b-kube-api-access-frqb7\" (UniqueName: \"kubernetes.io/projected/20147f52-0076-4eb0-aad7-5f262111a38b-kube-api-access-frqb7\") pod \"20147f52-0076-4eb0-aad7-5f262111a38b\" (UID: \"20147f52-0076-4eb0-aad7-5f262111a38b\") "
Oct 30 20:17:27.569870 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.569813    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20147f52-0076-4eb0-aad7-5f262111a38b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20147f52-0076-4eb0-aad7-5f262111a38b-plugins-config-volume\") pod \"20147f52-0076-4eb0-aad7-5f262111a38b\" (UID: \"20147f52-0076-4eb0-aad7-5f262111a38b\") "
Oct 30 20:17:27.570218 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.570181    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20147f52-0076-4eb0-aad7-5f262111a38b-plugins-config-volume" pod "20147f52-0076-4eb0-aad7-5f262111a38b" (UID: "20147f52-0076-4eb0-aad7-5f262111a38b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:27.570514 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.570494    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20147f52-0076-4eb0-aad7-5f262111a38b-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:27.570601 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.570521    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrtfb\" (UniqueName: \"kubernetes.io/projected/bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35-kube-api-access-zrtfb\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:27.572224 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.572200    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20147f52-0076-4eb0-aad7-5f262111a38b-kube-api-access-frqb7" pod "20147f52-0076-4eb0-aad7-5f262111a38b" (UID: "20147f52-0076-4eb0-aad7-5f262111a38b"). InnerVolumeSpecName "kube-api-access-frqb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:27.671799 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:27.671715    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frqb7\" (UniqueName: \"kubernetes.io/projected/20147f52-0076-4eb0-aad7-5f262111a38b-kube-api-access-frqb7\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:28.429368 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:28.429286    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"
Oct 30 20:17:28.429368 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:28.429302    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv" event={"ID":"bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35","Type":"ContainerDied","Data":"6ba740b5f831ea7b53bff5c2f489ff7643d4af1ab045b8f379ddf2d93aa1f27b"}
Oct 30 20:17:28.429368 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:28.429367    6030 scope.go:122] "RemoveContainer" containerID="7520eb36410c2ce41608bc2e4b0265e0958af5dfbec9bdc40a01097b1c76f0f1"
Oct 30 20:17:28.430859 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:28.430827    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"
Oct 30 20:17:28.430995 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:28.430854    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs" event={"ID":"20147f52-0076-4eb0-aad7-5f262111a38b","Type":"ContainerDied","Data":"3665bc50b01ef1c7c1c20c29e53a02af147bb73262aed31aa6992d4868b3d304"}
Oct 30 20:17:28.440154 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:28.440109    6030 scope.go:122] "RemoveContainer" containerID="e0c09b9c4034a92b473c1cc58e3c81d3352ac5ae3e5d3345b0a38670161170f5"
Oct 30 20:17:28.461194 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:28.461169    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"]
Oct 30 20:17:28.463639 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:28.463617    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b24a1a-7d8b65f6ff-ggxvs"]
Oct 30 20:17:28.480499 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:28.480475    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"]
Oct 30 20:17:28.484655 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:28.484631    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b24a1a-5bc559f94b-s84hv"]
Oct 30 20:17:29.203253 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:29.203218    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20147f52-0076-4eb0-aad7-5f262111a38b" path="/var/lib/kubelet/pods/20147f52-0076-4eb0-aad7-5f262111a38b/volumes"
Oct 30 20:17:29.203701 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:29.203687    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35" path="/var/lib/kubelet/pods/bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35/volumes"
Oct 30 20:17:30.055382 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.055343    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4" podUID="91e2a5df-6430-4eab-8959-ac0ca5a710ea" containerName="payload-processing" containerID="cri-o://fecd2c03b706292ec95cfe8731a520c8eb97cd2f2c63c3da55a23f8373dd5bff" gracePeriod=25
Oct 30 20:17:30.064623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.064576    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w" podUID="b71903da-bab1-4a53-9576-a7357dfd22df" containerName="payload-pre-processing" containerID="cri-o://803991ff564131928a97dce23bc13b42b6d31e37e4bd6d0ae95336f89dce134a" gracePeriod=25
Oct 30 20:17:30.449530 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.449493    6030 generic.go:356] "Generic (PLEG): container finished" podID="c75bdfb0-89bd-4935-baae-55475fdb405a" containerID="be0e816709503de473df06f1d40d141a1ce92f3db035d05f848040644c047e18" exitCode=0
Oct 30 20:17:30.449712 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.449560    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm" event={"ID":"c75bdfb0-89bd-4935-baae-55475fdb405a","Type":"ContainerDied","Data":"be0e816709503de473df06f1d40d141a1ce92f3db035d05f848040644c047e18"}
Oct 30 20:17:30.449712 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.449623    6030 scope.go:122] "RemoveContainer" containerID="80458fcd0987a05c5e049bbf172ce288143f933e723e9060d82b20e5789b24c2"
Oct 30 20:17:30.675297 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.675272    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 20:17:30.708812 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.708746    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75bdfb0-89bd-4935-baae-55475fdb405a-kube-api-access-wj4b2\" (UniqueName: \"kubernetes.io/projected/c75bdfb0-89bd-4935-baae-55475fdb405a-kube-api-access-wj4b2\") pod \"c75bdfb0-89bd-4935-baae-55475fdb405a\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") "
Oct 30 20:17:30.708812 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.708789    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-podinfo\") pod \"c75bdfb0-89bd-4935-baae-55475fdb405a\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") "
Oct 30 20:17:30.709065 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.708959    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-token\" (UniqueName: \"kubernetes.io/projected/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-token\") pod \"c75bdfb0-89bd-4935-baae-55475fdb405a\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") "
Oct 30 20:17:30.709065 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.708997    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-envoy\") pod \"c75bdfb0-89bd-4935-baae-55475fdb405a\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") "
Oct 30 20:17:30.709198 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.709137    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-credential-socket\") pod \"c75bdfb0-89bd-4935-baae-55475fdb405a\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") "
Oct 30 20:17:30.709265 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.709210    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-workload-certs\") pod \"c75bdfb0-89bd-4935-baae-55475fdb405a\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") "
Oct 30 20:17:30.709265 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.709241    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75bdfb0-89bd-4935-baae-55475fdb405a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75bdfb0-89bd-4935-baae-55475fdb405a-istiod-ca-cert\") pod \"c75bdfb0-89bd-4935-baae-55475fdb405a\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") "
Oct 30 20:17:30.709376 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.709274    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-workload-socket\") pod \"c75bdfb0-89bd-4935-baae-55475fdb405a\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") "
Oct 30 20:17:30.709376 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.709321    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-data\") pod \"c75bdfb0-89bd-4935-baae-55475fdb405a\" (UID: \"c75bdfb0-89bd-4935-baae-55475fdb405a\") "
Oct 30 20:17:30.709924 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.709873    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-credential-socket" pod "c75bdfb0-89bd-4935-baae-55475fdb405a" (UID: "c75bdfb0-89bd-4935-baae-55475fdb405a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:30.710136 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.710000    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-workload-certs" pod "c75bdfb0-89bd-4935-baae-55475fdb405a" (UID: "c75bdfb0-89bd-4935-baae-55475fdb405a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:30.710136 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.710004    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-data" pod "c75bdfb0-89bd-4935-baae-55475fdb405a" (UID: "c75bdfb0-89bd-4935-baae-55475fdb405a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:30.710136 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.710086    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75bdfb0-89bd-4935-baae-55475fdb405a-istiod-ca-cert" pod "c75bdfb0-89bd-4935-baae-55475fdb405a" (UID: "c75bdfb0-89bd-4935-baae-55475fdb405a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:30.710389 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.710208    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-workload-socket" pod "c75bdfb0-89bd-4935-baae-55475fdb405a" (UID: "c75bdfb0-89bd-4935-baae-55475fdb405a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:30.713012 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.712988    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-token" pod "c75bdfb0-89bd-4935-baae-55475fdb405a" (UID: "c75bdfb0-89bd-4935-baae-55475fdb405a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:30.713425 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.713391    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-podinfo" pod "c75bdfb0-89bd-4935-baae-55475fdb405a" (UID: "c75bdfb0-89bd-4935-baae-55475fdb405a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:17:30.713425 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.713395    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75bdfb0-89bd-4935-baae-55475fdb405a-kube-api-access-wj4b2" pod "c75bdfb0-89bd-4935-baae-55475fdb405a" (UID: "c75bdfb0-89bd-4935-baae-55475fdb405a"). InnerVolumeSpecName "kube-api-access-wj4b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:30.733359 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.733324    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-envoy" pod "c75bdfb0-89bd-4935-baae-55475fdb405a" (UID: "c75bdfb0-89bd-4935-baae-55475fdb405a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:30.810745 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.810710    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:30.810745 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.810740    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:30.810745 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.810750    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:30.810978 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.810761    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:30.810978 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.810770    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75bdfb0-89bd-4935-baae-55475fdb405a-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:30.810978 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.810778    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:30.810978 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.810786    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:30.810978 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.810826    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj4b2\" (UniqueName: \"kubernetes.io/projected/c75bdfb0-89bd-4935-baae-55475fdb405a-kube-api-access-wj4b2\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:30.810978 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:30.810836    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75bdfb0-89bd-4935-baae-55475fdb405a-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:31.456918 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:31.456884    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"
Oct 30 20:17:31.456918 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:31.456903    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm" event={"ID":"c75bdfb0-89bd-4935-baae-55475fdb405a","Type":"ContainerDied","Data":"a65ef055f13e1f2bb3af3aebfab1f83a38311ec69c79b27d6068581f9e8b097d"}
Oct 30 20:17:31.457506 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:31.456959    6030 scope.go:122] "RemoveContainer" containerID="be0e816709503de473df06f1d40d141a1ce92f3db035d05f848040644c047e18"
Oct 30 20:17:31.481714 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:31.481689    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"]
Oct 30 20:17:31.487361 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:31.487336    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-c2kzm"]
Oct 30 20:17:33.204419 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:33.204388    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75bdfb0-89bd-4935-baae-55475fdb405a" path="/var/lib/kubelet/pods/c75bdfb0-89bd-4935-baae-55475fdb405a/volumes"
Oct 30 20:17:34.288886 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:17:34.288853    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:17:35.820164 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:35.820121    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"]
Oct 30 20:17:35.857427 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:35.857390    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"]
Oct 30 20:17:36.126141 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:36.126109    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"]
Oct 30 20:17:36.224066 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:36.222431    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"]
Oct 30 20:17:36.641211 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:36.641175    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w" podUID="b71903da-bab1-4a53-9576-a7357dfd22df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:17:36.641211 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:36.641202    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4" podUID="91e2a5df-6430-4eab-8959-ac0ca5a710ea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:17:40.820805 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:40.820743    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph" podUID="1dfa9318-c8b7-4c06-91c5-f630b3dc7810" containerName="payload-processing" containerID="cri-o://7b124f7a513a4e71d4037e30a62a911a8cb2695e51a7f57738d379933bc43d1a" gracePeriod=25
Oct 30 20:17:40.858623 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:40.858565    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8" podUID="40974280-ed25-41d5-8697-4c09d86aadb1" containerName="payload-pre-processing" containerID="cri-o://b98c2d919b7f4e0e66f3469c1b01e04fab609ceb8c22748f58e764f6657082c6" gracePeriod=25
Oct 30 20:17:41.126391 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.126356    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68" podUID="f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4" containerName="payload-processing" containerID="cri-o://fdcb336a5775068b24f226049a2e95ba9c5995a06d9ba05fad312182afb11355" gracePeriod=25
Oct 30 20:17:41.223193 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.223141    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q" podUID="a414b711-860e-4c19-be75-4c1c1abc70b8" containerName="payload-pre-processing" containerID="cri-o://9603e6d1bac62e61b2e1b5ebccbc5f585a73d2f937c42cea9514c3d525b25397" gracePeriod=25
Oct 30 20:17:41.421982 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.421949    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"
Oct 30 20:17:41.518595 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.518566    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"
Oct 30 20:17:41.540012 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.539980    6030 generic.go:356] "Generic (PLEG): container finished" podID="a414b711-860e-4c19-be75-4c1c1abc70b8" containerID="9603e6d1bac62e61b2e1b5ebccbc5f585a73d2f937c42cea9514c3d525b25397" exitCode=0
Oct 30 20:17:41.540191 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.540022    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"
Oct 30 20:17:41.540191 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.540092    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q" event={"ID":"a414b711-860e-4c19-be75-4c1c1abc70b8","Type":"ContainerDied","Data":"9603e6d1bac62e61b2e1b5ebccbc5f585a73d2f937c42cea9514c3d525b25397"}
Oct 30 20:17:41.540191 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.540139    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q" event={"ID":"a414b711-860e-4c19-be75-4c1c1abc70b8","Type":"ContainerDied","Data":"cc633dbcd6c35a93245ea561eee93d85e740bdeb5ab32e3c6a9443c005f295f3"}
Oct 30 20:17:41.540191 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.540169    6030 scope.go:122] "RemoveContainer" containerID="9603e6d1bac62e61b2e1b5ebccbc5f585a73d2f937c42cea9514c3d525b25397"
Oct 30 20:17:41.542418 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.542372    6030 generic.go:356] "Generic (PLEG): container finished" podID="f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4" containerID="fdcb336a5775068b24f226049a2e95ba9c5995a06d9ba05fad312182afb11355" exitCode=0
Oct 30 20:17:41.542418 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.542406    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68" event={"ID":"f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4","Type":"ContainerDied","Data":"fdcb336a5775068b24f226049a2e95ba9c5995a06d9ba05fad312182afb11355"}
Oct 30 20:17:41.542659 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.542421    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"
Oct 30 20:17:41.542659 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.542449    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68" event={"ID":"f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4","Type":"ContainerDied","Data":"e93d9d60972c9a1aa66e96e1f8fd62db8eaabcbeb1f799d64aaac3b8d30803d1"}
Oct 30 20:17:41.552574 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.552552    6030 scope.go:122] "RemoveContainer" containerID="9603e6d1bac62e61b2e1b5ebccbc5f585a73d2f937c42cea9514c3d525b25397"
Oct 30 20:17:41.552919 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:17:41.552891    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9603e6d1bac62e61b2e1b5ebccbc5f585a73d2f937c42cea9514c3d525b25397\": container with ID starting with 9603e6d1bac62e61b2e1b5ebccbc5f585a73d2f937c42cea9514c3d525b25397 not found: ID does not exist" containerID="9603e6d1bac62e61b2e1b5ebccbc5f585a73d2f937c42cea9514c3d525b25397"
Oct 30 20:17:41.553008 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.552928    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9603e6d1bac62e61b2e1b5ebccbc5f585a73d2f937c42cea9514c3d525b25397"} err="failed to get container status \"9603e6d1bac62e61b2e1b5ebccbc5f585a73d2f937c42cea9514c3d525b25397\": rpc error: code = NotFound desc = could not find container \"9603e6d1bac62e61b2e1b5ebccbc5f585a73d2f937c42cea9514c3d525b25397\": container with ID starting with 9603e6d1bac62e61b2e1b5ebccbc5f585a73d2f937c42cea9514c3d525b25397 not found: ID does not exist"
Oct 30 20:17:41.553008 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.552953    6030 scope.go:122] "RemoveContainer" containerID="fdcb336a5775068b24f226049a2e95ba9c5995a06d9ba05fad312182afb11355"
Oct 30 20:17:41.558539 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.558510    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4-plugins-config-volume\") pod \"f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4\" (UID: \"f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4\") "
Oct 30 20:17:41.558659 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.558632    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4-kube-api-access-v4rwk\" (UniqueName: \"kubernetes.io/projected/f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4-kube-api-access-v4rwk\") pod \"f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4\" (UID: \"f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4\") "
Oct 30 20:17:41.558924 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.558893    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4-plugins-config-volume" pod "f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4" (UID: "f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:41.559317 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.559279    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:41.560943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.560917    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4-kube-api-access-v4rwk" pod "f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4" (UID: "f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4"). InnerVolumeSpecName "kube-api-access-v4rwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:41.568229 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.568203    6030 scope.go:122] "RemoveContainer" containerID="fdcb336a5775068b24f226049a2e95ba9c5995a06d9ba05fad312182afb11355"
Oct 30 20:17:41.568578 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:17:41.568557    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdcb336a5775068b24f226049a2e95ba9c5995a06d9ba05fad312182afb11355\": container with ID starting with fdcb336a5775068b24f226049a2e95ba9c5995a06d9ba05fad312182afb11355 not found: ID does not exist" containerID="fdcb336a5775068b24f226049a2e95ba9c5995a06d9ba05fad312182afb11355"
Oct 30 20:17:41.568657 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.568588    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdcb336a5775068b24f226049a2e95ba9c5995a06d9ba05fad312182afb11355"} err="failed to get container status \"fdcb336a5775068b24f226049a2e95ba9c5995a06d9ba05fad312182afb11355\": rpc error: code = NotFound desc = could not find container \"fdcb336a5775068b24f226049a2e95ba9c5995a06d9ba05fad312182afb11355\": container with ID starting with fdcb336a5775068b24f226049a2e95ba9c5995a06d9ba05fad312182afb11355 not found: ID does not exist"
Oct 30 20:17:41.660791 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.660704    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a414b711-860e-4c19-be75-4c1c1abc70b8-kube-api-access-4swbm\" (UniqueName: \"kubernetes.io/projected/a414b711-860e-4c19-be75-4c1c1abc70b8-kube-api-access-4swbm\") pod \"a414b711-860e-4c19-be75-4c1c1abc70b8\" (UID: \"a414b711-860e-4c19-be75-4c1c1abc70b8\") "
Oct 30 20:17:41.660964 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.660873    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a414b711-860e-4c19-be75-4c1c1abc70b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a414b711-860e-4c19-be75-4c1c1abc70b8-plugins-config-volume\") pod \"a414b711-860e-4c19-be75-4c1c1abc70b8\" (UID: \"a414b711-860e-4c19-be75-4c1c1abc70b8\") "
Oct 30 20:17:41.661374 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.661339    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a414b711-860e-4c19-be75-4c1c1abc70b8-plugins-config-volume" pod "a414b711-860e-4c19-be75-4c1c1abc70b8" (UID: "a414b711-860e-4c19-be75-4c1c1abc70b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:41.661524 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.661347    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4rwk\" (UniqueName: \"kubernetes.io/projected/f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4-kube-api-access-v4rwk\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:41.663150 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.663125    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a414b711-860e-4c19-be75-4c1c1abc70b8-kube-api-access-4swbm" pod "a414b711-860e-4c19-be75-4c1c1abc70b8" (UID: "a414b711-860e-4c19-be75-4c1c1abc70b8"). InnerVolumeSpecName "kube-api-access-4swbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:41.765068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.762772    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4swbm\" (UniqueName: \"kubernetes.io/projected/a414b711-860e-4c19-be75-4c1c1abc70b8-kube-api-access-4swbm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:41.765068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.762809    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a414b711-860e-4c19-be75-4c1c1abc70b8-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:41.870379 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.870342    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"]
Oct 30 20:17:41.880640 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.880608    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-54d4c4-5bc9c76fff-hzg9q"]
Oct 30 20:17:41.893337 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.893306    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"]
Oct 30 20:17:41.900912 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:41.900882    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-54d4c4-79d84c6674-jvq68"]
Oct 30 20:17:43.209742 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:43.209707    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a414b711-860e-4c19-be75-4c1c1abc70b8" path="/var/lib/kubelet/pods/a414b711-860e-4c19-be75-4c1c1abc70b8/volumes"
Oct 30 20:17:43.210203 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:43.210187    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4" path="/var/lib/kubelet/pods/f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4/volumes"
Oct 30 20:17:46.641343 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:46.641303    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4" podUID="91e2a5df-6430-4eab-8959-ac0ca5a710ea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:17:46.641343 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:46.641324    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w" podUID="b71903da-bab1-4a53-9576-a7357dfd22df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:17:48.342860 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:48.342821    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph" podUID="1dfa9318-c8b7-4c06-91c5-f630b3dc7810" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:17:48.343545 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:48.343509    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8" podUID="40974280-ed25-41d5-8697-4c09d86aadb1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:17:50.350725 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.350688    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"
Oct 30 20:17:50.378507 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.378460    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91e2a5df-6430-4eab-8959-ac0ca5a710ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e2a5df-6430-4eab-8959-ac0ca5a710ea-plugins-config-volume\") pod \"91e2a5df-6430-4eab-8959-ac0ca5a710ea\" (UID: \"91e2a5df-6430-4eab-8959-ac0ca5a710ea\") "
Oct 30 20:17:50.378685 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.378624    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e2a5df-6430-4eab-8959-ac0ca5a710ea-kube-api-access-xhxsk\" (UniqueName: \"kubernetes.io/projected/91e2a5df-6430-4eab-8959-ac0ca5a710ea-kube-api-access-xhxsk\") pod \"91e2a5df-6430-4eab-8959-ac0ca5a710ea\" (UID: \"91e2a5df-6430-4eab-8959-ac0ca5a710ea\") "
Oct 30 20:17:50.379183 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.379151    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e2a5df-6430-4eab-8959-ac0ca5a710ea-plugins-config-volume" pod "91e2a5df-6430-4eab-8959-ac0ca5a710ea" (UID: "91e2a5df-6430-4eab-8959-ac0ca5a710ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:50.382335 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.382305    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e2a5df-6430-4eab-8959-ac0ca5a710ea-kube-api-access-xhxsk" pod "91e2a5df-6430-4eab-8959-ac0ca5a710ea" (UID: "91e2a5df-6430-4eab-8959-ac0ca5a710ea"). InnerVolumeSpecName "kube-api-access-xhxsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:50.416793 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.416767    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"
Oct 30 20:17:50.479848 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.479817    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b71903da-bab1-4a53-9576-a7357dfd22df-kube-api-access-t5zjh\" (UniqueName: \"kubernetes.io/projected/b71903da-bab1-4a53-9576-a7357dfd22df-kube-api-access-t5zjh\") pod \"b71903da-bab1-4a53-9576-a7357dfd22df\" (UID: \"b71903da-bab1-4a53-9576-a7357dfd22df\") "
Oct 30 20:17:50.480021 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.479997    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b71903da-bab1-4a53-9576-a7357dfd22df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71903da-bab1-4a53-9576-a7357dfd22df-plugins-config-volume\") pod \"b71903da-bab1-4a53-9576-a7357dfd22df\" (UID: \"b71903da-bab1-4a53-9576-a7357dfd22df\") "
Oct 30 20:17:50.480448 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.480421    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71903da-bab1-4a53-9576-a7357dfd22df-plugins-config-volume" pod "b71903da-bab1-4a53-9576-a7357dfd22df" (UID: "b71903da-bab1-4a53-9576-a7357dfd22df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:50.480548 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.480447    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhxsk\" (UniqueName: \"kubernetes.io/projected/91e2a5df-6430-4eab-8959-ac0ca5a710ea-kube-api-access-xhxsk\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:50.480548 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.480469    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e2a5df-6430-4eab-8959-ac0ca5a710ea-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:50.482302 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.482271    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71903da-bab1-4a53-9576-a7357dfd22df-kube-api-access-t5zjh" pod "b71903da-bab1-4a53-9576-a7357dfd22df" (UID: "b71903da-bab1-4a53-9576-a7357dfd22df"). InnerVolumeSpecName "kube-api-access-t5zjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:50.581232 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.581151    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71903da-bab1-4a53-9576-a7357dfd22df-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:50.581232 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.581180    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5zjh\" (UniqueName: \"kubernetes.io/projected/b71903da-bab1-4a53-9576-a7357dfd22df-kube-api-access-t5zjh\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:50.630075 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.630009    6030 generic.go:356] "Generic (PLEG): container finished" podID="91e2a5df-6430-4eab-8959-ac0ca5a710ea" containerID="fecd2c03b706292ec95cfe8731a520c8eb97cd2f2c63c3da55a23f8373dd5bff" exitCode=0
Oct 30 20:17:50.630284 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.630086    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"
Oct 30 20:17:50.630284 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.630094    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4" event={"ID":"91e2a5df-6430-4eab-8959-ac0ca5a710ea","Type":"ContainerDied","Data":"fecd2c03b706292ec95cfe8731a520c8eb97cd2f2c63c3da55a23f8373dd5bff"}
Oct 30 20:17:50.630284 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.630181    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4" event={"ID":"91e2a5df-6430-4eab-8959-ac0ca5a710ea","Type":"ContainerDied","Data":"3e6a93b3aaa311b276e6e171c738c4038c295e84557ce6aec5dfb75ea37e6131"}
Oct 30 20:17:50.630284 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.630214    6030 scope.go:122] "RemoveContainer" containerID="fecd2c03b706292ec95cfe8731a520c8eb97cd2f2c63c3da55a23f8373dd5bff"
Oct 30 20:17:50.631953 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.631932    6030 generic.go:356] "Generic (PLEG): container finished" podID="b71903da-bab1-4a53-9576-a7357dfd22df" containerID="803991ff564131928a97dce23bc13b42b6d31e37e4bd6d0ae95336f89dce134a" exitCode=0
Oct 30 20:17:50.632095 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.631985    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"
Oct 30 20:17:50.632095 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.631999    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w" event={"ID":"b71903da-bab1-4a53-9576-a7357dfd22df","Type":"ContainerDied","Data":"803991ff564131928a97dce23bc13b42b6d31e37e4bd6d0ae95336f89dce134a"}
Oct 30 20:17:50.632095 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.632036    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w" event={"ID":"b71903da-bab1-4a53-9576-a7357dfd22df","Type":"ContainerDied","Data":"4eee9765c69ad8aaa1e00e30efab81233f9b16105cc527ec44e4f55404d19cc6"}
Oct 30 20:17:50.643598 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.643466    6030 scope.go:122] "RemoveContainer" containerID="fecd2c03b706292ec95cfe8731a520c8eb97cd2f2c63c3da55a23f8373dd5bff"
Oct 30 20:17:50.643805 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:17:50.643785    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fecd2c03b706292ec95cfe8731a520c8eb97cd2f2c63c3da55a23f8373dd5bff\": container with ID starting with fecd2c03b706292ec95cfe8731a520c8eb97cd2f2c63c3da55a23f8373dd5bff not found: ID does not exist" containerID="fecd2c03b706292ec95cfe8731a520c8eb97cd2f2c63c3da55a23f8373dd5bff"
Oct 30 20:17:50.643854 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.643817    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fecd2c03b706292ec95cfe8731a520c8eb97cd2f2c63c3da55a23f8373dd5bff"} err="failed to get container status \"fecd2c03b706292ec95cfe8731a520c8eb97cd2f2c63c3da55a23f8373dd5bff\": rpc error: code = NotFound desc = could not find container \"fecd2c03b706292ec95cfe8731a520c8eb97cd2f2c63c3da55a23f8373dd5bff\": container with ID starting with fecd2c03b706292ec95cfe8731a520c8eb97cd2f2c63c3da55a23f8373dd5bff not found: ID does not exist"
Oct 30 20:17:50.643854 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.643832    6030 scope.go:122] "RemoveContainer" containerID="803991ff564131928a97dce23bc13b42b6d31e37e4bd6d0ae95336f89dce134a"
Oct 30 20:17:50.654858 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.654833    6030 scope.go:122] "RemoveContainer" containerID="803991ff564131928a97dce23bc13b42b6d31e37e4bd6d0ae95336f89dce134a"
Oct 30 20:17:50.655313 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:17:50.655280    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"803991ff564131928a97dce23bc13b42b6d31e37e4bd6d0ae95336f89dce134a\": container with ID starting with 803991ff564131928a97dce23bc13b42b6d31e37e4bd6d0ae95336f89dce134a not found: ID does not exist" containerID="803991ff564131928a97dce23bc13b42b6d31e37e4bd6d0ae95336f89dce134a"
Oct 30 20:17:50.655491 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.655321    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803991ff564131928a97dce23bc13b42b6d31e37e4bd6d0ae95336f89dce134a"} err="failed to get container status \"803991ff564131928a97dce23bc13b42b6d31e37e4bd6d0ae95336f89dce134a\": rpc error: code = NotFound desc = could not find container \"803991ff564131928a97dce23bc13b42b6d31e37e4bd6d0ae95336f89dce134a\": container with ID starting with 803991ff564131928a97dce23bc13b42b6d31e37e4bd6d0ae95336f89dce134a not found: ID does not exist"
Oct 30 20:17:50.658406 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.658384    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"]
Oct 30 20:17:50.667896 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.667871    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7cbcbb-69f8f4d74d-q8vd4"]
Oct 30 20:17:50.681738 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.681710    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"]
Oct 30 20:17:50.685434 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:50.685412    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7cbcbb-678bf465f-hwx2w"]
Oct 30 20:17:51.201500 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:51.201465    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e2a5df-6430-4eab-8959-ac0ca5a710ea" path="/var/lib/kubelet/pods/91e2a5df-6430-4eab-8959-ac0ca5a710ea/volumes"
Oct 30 20:17:51.201881 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:51.201868    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b71903da-bab1-4a53-9576-a7357dfd22df" path="/var/lib/kubelet/pods/b71903da-bab1-4a53-9576-a7357dfd22df/volumes"
Oct 30 20:17:58.341885 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:58.341844    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph" podUID="1dfa9318-c8b7-4c06-91c5-f630b3dc7810" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:17:58.341885 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:58.341876    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8" podUID="40974280-ed25-41d5-8697-4c09d86aadb1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:17:59.044357 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:59.044323    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"]
Oct 30 20:17:59.051526 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:59.051495    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"]
Oct 30 20:17:59.066485 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:59.066453    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"]
Oct 30 20:17:59.081229 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:17:59.081197    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"]
Oct 30 20:18:00.946715 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.946666    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"]
Oct 30 20:18:00.947480 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947454    6030 cpu_manager.go:414] "containerMap: removing container" podUID="a414b711-860e-4c19-be75-4c1c1abc70b8" containerName="payload-pre-processing"
Oct 30 20:18:00.947480 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947480    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a414b711-860e-4c19-be75-4c1c1abc70b8" containerName="payload-pre-processing"
Oct 30 20:18:00.947654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947495    6030 cpu_manager.go:414] "containerMap: removing container" podUID="8790b73a-43cc-496c-ab9e-250aabb2306f" containerName="maas-api"
Oct 30 20:18:00.947654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947502    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8790b73a-43cc-496c-ab9e-250aabb2306f" containerName="maas-api"
Oct 30 20:18:00.947654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947516    6030 cpu_manager.go:414] "containerMap: removing container" podUID="c75bdfb0-89bd-4935-baae-55475fdb405a" containerName="istio-proxy"
Oct 30 20:18:00.947654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947524    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75bdfb0-89bd-4935-baae-55475fdb405a" containerName="istio-proxy"
Oct 30 20:18:00.947654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947536    6030 cpu_manager.go:414] "containerMap: removing container" podUID="0118f805-4deb-4de0-aaf2-cad5ba1ed2fa" containerName="payload-pre-processing"
Oct 30 20:18:00.947654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947543    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0118f805-4deb-4de0-aaf2-cad5ba1ed2fa" containerName="payload-pre-processing"
Oct 30 20:18:00.947654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947570    6030 cpu_manager.go:414] "containerMap: removing container" podUID="b71903da-bab1-4a53-9576-a7357dfd22df" containerName="payload-pre-processing"
Oct 30 20:18:00.947654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947578    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b71903da-bab1-4a53-9576-a7357dfd22df" containerName="payload-pre-processing"
Oct 30 20:18:00.947654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947587    6030 cpu_manager.go:414] "containerMap: removing container" podUID="0b80bf01-a7f4-41ae-9adc-f550c6ec7707" containerName="payload-processing"
Oct 30 20:18:00.947654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947594    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b80bf01-a7f4-41ae-9adc-f550c6ec7707" containerName="payload-processing"
Oct 30 20:18:00.947654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947616    6030 cpu_manager.go:414] "containerMap: removing container" podUID="91e2a5df-6430-4eab-8959-ac0ca5a710ea" containerName="payload-processing"
Oct 30 20:18:00.947654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947624    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e2a5df-6430-4eab-8959-ac0ca5a710ea" containerName="payload-processing"
Oct 30 20:18:00.947654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947646    6030 cpu_manager.go:414] "containerMap: removing container" podUID="bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35" containerName="payload-processing"
Oct 30 20:18:00.947654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947653    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35" containerName="payload-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947662    6030 cpu_manager.go:414] "containerMap: removing container" podUID="3fbd9f0c-7c56-406e-8dd9-0770f49f4878" containerName="payload-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947671    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fbd9f0c-7c56-406e-8dd9-0770f49f4878" containerName="payload-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947691    6030 cpu_manager.go:414] "containerMap: removing container" podUID="f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4" containerName="payload-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947698    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4" containerName="payload-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947709    6030 cpu_manager.go:414] "containerMap: removing container" podUID="aea211f6-4038-4d8f-a09f-581a68b6c73d" containerName="payload-pre-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947716    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea211f6-4038-4d8f-a09f-581a68b6c73d" containerName="payload-pre-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947736    6030 cpu_manager.go:414] "containerMap: removing container" podUID="20147f52-0076-4eb0-aad7-5f262111a38b" containerName="payload-pre-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947743    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20147f52-0076-4eb0-aad7-5f262111a38b" containerName="payload-pre-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947896    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="b71903da-bab1-4a53-9576-a7357dfd22df" containerName="payload-pre-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947910    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b80bf01-a7f4-41ae-9adc-f550c6ec7707" containerName="payload-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947922    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e2a5df-6430-4eab-8959-ac0ca5a710ea" containerName="payload-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947933    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd0e09f-6e9a-4bac-bf50-890f7e6e9a35" containerName="payload-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947942    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fbd9f0c-7c56-406e-8dd9-0770f49f4878" containerName="payload-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947953    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="f35bc1c5-b72c-4e13-a28a-80b1f9e4bdf4" containerName="payload-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947963    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea211f6-4038-4d8f-a09f-581a68b6c73d" containerName="payload-pre-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947973    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75bdfb0-89bd-4935-baae-55475fdb405a" containerName="istio-proxy"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947981    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="20147f52-0076-4eb0-aad7-5f262111a38b" containerName="payload-pre-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.947994    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="a414b711-860e-4c19-be75-4c1c1abc70b8" containerName="payload-pre-processing"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.948002    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="8790b73a-43cc-496c-ab9e-250aabb2306f" containerName="maas-api"
Oct 30 20:18:00.948446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.948013    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="0118f805-4deb-4de0-aaf2-cad5ba1ed2fa" containerName="payload-pre-processing"
Oct 30 20:18:00.952209 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.951823    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:18:00.965811 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.965128    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-89c124-serving-cert\""
Oct 30 20:18:00.974296 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:00.973576    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"]
Oct 30 20:18:01.012736 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.012304    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-metrics-certs\") pod \"maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz\" (UID: \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:18:01.012736 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.012370    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-openshift-service-ca\") pod \"maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz\" (UID: \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:18:01.012736 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.012419    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-maas-api-tls\") pod \"maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz\" (UID: \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:18:01.012736 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.012468    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54z2z\" (UniqueName: \"kubernetes.io/projected/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-kube-api-access-54z2z\") pod \"maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz\" (UID: \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:18:01.115670 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.114597    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-metrics-certs\") pod \"maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz\" (UID: \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:18:01.115670 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.114651    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-openshift-service-ca\") pod \"maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz\" (UID: \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:18:01.115670 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.114709    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-maas-api-tls\") pod \"maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz\" (UID: \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:18:01.115670 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.114755    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54z2z\" (UniqueName: \"kubernetes.io/projected/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-kube-api-access-54z2z\") pod \"maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz\" (UID: \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:18:01.115670 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:18:01.115368    6030 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-serving-cert: secret "maas-api-e2e-inf-a-89c124-serving-cert" not found
Oct 30 20:18:01.115670 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:18:01.115432    6030 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-maas-api-tls podName:0f6f8eab-4484-4542-9e49-9b13bb4a8cef nodeName:}" failed. No retries permitted until 2026-10-30 20:18:01.615412847 +0000 UTC m=+2760.531019701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-maas-api-tls") pod "maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz" (UID: "0f6f8eab-4484-4542-9e49-9b13bb4a8cef") : secret "maas-api-e2e-inf-a-89c124-serving-cert" not found
Oct 30 20:18:01.117142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.115805    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-openshift-service-ca\") pod \"maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz\" (UID: \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:18:01.144786 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.143883    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-metrics-certs\") pod \"maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz\" (UID: \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:18:01.147809 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.147756    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54z2z\" (UniqueName: \"kubernetes.io/projected/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-kube-api-access-54z2z\") pod \"maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz\" (UID: \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:18:01.180316 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.180291    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"
Oct 30 20:18:01.188479 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.188458    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"
Oct 30 20:18:01.216529 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.216488    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dfa9318-c8b7-4c06-91c5-f630b3dc7810-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfa9318-c8b7-4c06-91c5-f630b3dc7810-plugins-config-volume\") pod \"1dfa9318-c8b7-4c06-91c5-f630b3dc7810\" (UID: \"1dfa9318-c8b7-4c06-91c5-f630b3dc7810\") "
Oct 30 20:18:01.216693 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.216613    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40974280-ed25-41d5-8697-4c09d86aadb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40974280-ed25-41d5-8697-4c09d86aadb1-plugins-config-volume\") pod \"40974280-ed25-41d5-8697-4c09d86aadb1\" (UID: \"40974280-ed25-41d5-8697-4c09d86aadb1\") "
Oct 30 20:18:01.216758 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.216692    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dfa9318-c8b7-4c06-91c5-f630b3dc7810-kube-api-access-tpdrb\" (UniqueName: \"kubernetes.io/projected/1dfa9318-c8b7-4c06-91c5-f630b3dc7810-kube-api-access-tpdrb\") pod \"1dfa9318-c8b7-4c06-91c5-f630b3dc7810\" (UID: \"1dfa9318-c8b7-4c06-91c5-f630b3dc7810\") "
Oct 30 20:18:01.216821 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.216767    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40974280-ed25-41d5-8697-4c09d86aadb1-kube-api-access-sxptl\" (UniqueName: \"kubernetes.io/projected/40974280-ed25-41d5-8697-4c09d86aadb1-kube-api-access-sxptl\") pod \"40974280-ed25-41d5-8697-4c09d86aadb1\" (UID: \"40974280-ed25-41d5-8697-4c09d86aadb1\") "
Oct 30 20:18:01.219100 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.219074    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40974280-ed25-41d5-8697-4c09d86aadb1-plugins-config-volume" pod "40974280-ed25-41d5-8697-4c09d86aadb1" (UID: "40974280-ed25-41d5-8697-4c09d86aadb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:01.219296 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.219272    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dfa9318-c8b7-4c06-91c5-f630b3dc7810-plugins-config-volume" pod "1dfa9318-c8b7-4c06-91c5-f630b3dc7810" (UID: "1dfa9318-c8b7-4c06-91c5-f630b3dc7810"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:01.220285 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.220255    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40974280-ed25-41d5-8697-4c09d86aadb1-kube-api-access-sxptl" pod "40974280-ed25-41d5-8697-4c09d86aadb1" (UID: "40974280-ed25-41d5-8697-4c09d86aadb1"). InnerVolumeSpecName "kube-api-access-sxptl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:01.221113 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.221084    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfa9318-c8b7-4c06-91c5-f630b3dc7810-kube-api-access-tpdrb" pod "1dfa9318-c8b7-4c06-91c5-f630b3dc7810" (UID: "1dfa9318-c8b7-4c06-91c5-f630b3dc7810"). InnerVolumeSpecName "kube-api-access-tpdrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:01.319564 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.319525    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxptl\" (UniqueName: \"kubernetes.io/projected/40974280-ed25-41d5-8697-4c09d86aadb1-kube-api-access-sxptl\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:01.319564 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.319557    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfa9318-c8b7-4c06-91c5-f630b3dc7810-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:01.319564 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.319567    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40974280-ed25-41d5-8697-4c09d86aadb1-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:01.319799 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.319575    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpdrb\" (UniqueName: \"kubernetes.io/projected/1dfa9318-c8b7-4c06-91c5-f630b3dc7810-kube-api-access-tpdrb\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:01.625481 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.625444    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-maas-api-tls\") pod \"maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz\" (UID: \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:18:01.634676 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.634641    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-maas-api-tls\") pod \"maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz\" (UID: \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:18:01.728465 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.728427    6030 generic.go:356] "Generic (PLEG): container finished" podID="1dfa9318-c8b7-4c06-91c5-f630b3dc7810" containerID="7b124f7a513a4e71d4037e30a62a911a8cb2695e51a7f57738d379933bc43d1a" exitCode=0
Oct 30 20:18:01.728661 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.728492    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"
Oct 30 20:18:01.728661 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.728513    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph" event={"ID":"1dfa9318-c8b7-4c06-91c5-f630b3dc7810","Type":"ContainerDied","Data":"7b124f7a513a4e71d4037e30a62a911a8cb2695e51a7f57738d379933bc43d1a"}
Oct 30 20:18:01.728661 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.728573    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph" event={"ID":"1dfa9318-c8b7-4c06-91c5-f630b3dc7810","Type":"ContainerDied","Data":"fe9d0702ea339e4020f00aeff2913e4cfb2b5d55e6d12a7ccd66f542b36cd598"}
Oct 30 20:18:01.728661 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.728595    6030 scope.go:122] "RemoveContainer" containerID="7b124f7a513a4e71d4037e30a62a911a8cb2695e51a7f57738d379933bc43d1a"
Oct 30 20:18:01.730099 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.730075    6030 generic.go:356] "Generic (PLEG): container finished" podID="40974280-ed25-41d5-8697-4c09d86aadb1" containerID="b98c2d919b7f4e0e66f3469c1b01e04fab609ceb8c22748f58e764f6657082c6" exitCode=0
Oct 30 20:18:01.730250 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.730117    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"
Oct 30 20:18:01.730250 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.730163    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8" event={"ID":"40974280-ed25-41d5-8697-4c09d86aadb1","Type":"ContainerDied","Data":"b98c2d919b7f4e0e66f3469c1b01e04fab609ceb8c22748f58e764f6657082c6"}
Oct 30 20:18:01.730250 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.730211    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8" event={"ID":"40974280-ed25-41d5-8697-4c09d86aadb1","Type":"ContainerDied","Data":"39b0a163815706f9e9bd467317ee4923f8d78e0087d63d8737979bdf1c470ad4"}
Oct 30 20:18:01.741209 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.740916    6030 scope.go:122] "RemoveContainer" containerID="7b124f7a513a4e71d4037e30a62a911a8cb2695e51a7f57738d379933bc43d1a"
Oct 30 20:18:01.741294 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:18:01.741224    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b124f7a513a4e71d4037e30a62a911a8cb2695e51a7f57738d379933bc43d1a\": container with ID starting with 7b124f7a513a4e71d4037e30a62a911a8cb2695e51a7f57738d379933bc43d1a not found: ID does not exist" containerID="7b124f7a513a4e71d4037e30a62a911a8cb2695e51a7f57738d379933bc43d1a"
Oct 30 20:18:01.741294 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.741257    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b124f7a513a4e71d4037e30a62a911a8cb2695e51a7f57738d379933bc43d1a"} err="failed to get container status \"7b124f7a513a4e71d4037e30a62a911a8cb2695e51a7f57738d379933bc43d1a\": rpc error: code = NotFound desc = could not find container \"7b124f7a513a4e71d4037e30a62a911a8cb2695e51a7f57738d379933bc43d1a\": container with ID starting with 7b124f7a513a4e71d4037e30a62a911a8cb2695e51a7f57738d379933bc43d1a not found: ID does not exist"
Oct 30 20:18:01.741294 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.741282    6030 scope.go:122] "RemoveContainer" containerID="b98c2d919b7f4e0e66f3469c1b01e04fab609ceb8c22748f58e764f6657082c6"
Oct 30 20:18:01.750722 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.750699    6030 scope.go:122] "RemoveContainer" containerID="b98c2d919b7f4e0e66f3469c1b01e04fab609ceb8c22748f58e764f6657082c6"
Oct 30 20:18:01.750972 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:18:01.750954    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98c2d919b7f4e0e66f3469c1b01e04fab609ceb8c22748f58e764f6657082c6\": container with ID starting with b98c2d919b7f4e0e66f3469c1b01e04fab609ceb8c22748f58e764f6657082c6 not found: ID does not exist" containerID="b98c2d919b7f4e0e66f3469c1b01e04fab609ceb8c22748f58e764f6657082c6"
Oct 30 20:18:01.751034 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.750977    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98c2d919b7f4e0e66f3469c1b01e04fab609ceb8c22748f58e764f6657082c6"} err="failed to get container status \"b98c2d919b7f4e0e66f3469c1b01e04fab609ceb8c22748f58e764f6657082c6\": rpc error: code = NotFound desc = could not find container \"b98c2d919b7f4e0e66f3469c1b01e04fab609ceb8c22748f58e764f6657082c6\": container with ID starting with b98c2d919b7f4e0e66f3469c1b01e04fab609ceb8c22748f58e764f6657082c6 not found: ID does not exist"
Oct 30 20:18:01.756383 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.756362    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"]
Oct 30 20:18:01.764558 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.764534    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-779fa3-7bcf9b566-59tph"]
Oct 30 20:18:01.784779 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.784745    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"]
Oct 30 20:18:01.788557 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.788526    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-779fa3-848d5f4b47-z8lw8"]
Oct 30 20:18:01.872830 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:01.872805    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:18:02.018418 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:02.018388    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"]
Oct 30 20:18:02.034826 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:02.034795    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"]
Oct 30 20:18:02.195723 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:02.195693    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"]
Oct 30 20:18:02.196281 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:18:02.196253    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6f8eab_4484_4542_9e49_9b13bb4a8cef.slice/crio-9c82851b7e7b92a3660e23f123aa6899472b2affe4c47b16263e0f358cc381a2 WatchSource:0}: Error finding container 9c82851b7e7b92a3660e23f123aa6899472b2affe4c47b16263e0f358cc381a2: Status 404 returned error can't find the container with id 9c82851b7e7b92a3660e23f123aa6899472b2affe4c47b16263e0f358cc381a2
Oct 30 20:18:02.198164 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:02.198143    6030 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:18:02.740097 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:02.740062    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz" event={"ID":"0f6f8eab-4484-4542-9e49-9b13bb4a8cef","Type":"ContainerStarted","Data":"9c82851b7e7b92a3660e23f123aa6899472b2affe4c47b16263e0f358cc381a2"}
Oct 30 20:18:03.202901 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:03.202866    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dfa9318-c8b7-4c06-91c5-f630b3dc7810" path="/var/lib/kubelet/pods/1dfa9318-c8b7-4c06-91c5-f630b3dc7810/volumes"
Oct 30 20:18:03.203425 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:03.203406    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40974280-ed25-41d5-8697-4c09d86aadb1" path="/var/lib/kubelet/pods/40974280-ed25-41d5-8697-4c09d86aadb1/volumes"
Oct 30 20:18:03.747305 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:03.747268    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz" event={"ID":"0f6f8eab-4484-4542-9e49-9b13bb4a8cef","Type":"ContainerStarted","Data":"779529070ea31bb9a6bec4dc8e8225766c36f6e1e7025b06128f9e8bc31b1058"}
Oct 30 20:18:03.747489 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:03.747468    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:18:03.771715 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:03.771662    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz" podStartSLOduration=3.297198277 podStartE2EDuration="3.771647352s" totalImagesPullingTime="474.449075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:03.769915924 +0000 UTC m=+2762.685522781" watchObservedRunningTime="2026-10-30 20:18:03.771647352 +0000 UTC m=+2762.687254208"
Oct 30 20:18:04.288096 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:18:04.288025    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:18:09.762611 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:09.762582    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:18:14.068138 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.068094    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" containerID="cri-o://8cfb8bc3d11d8b4c7a8b51f3b4b68a13b44f47872dd11af4274a347e16b5a20f" gracePeriod=45
Oct 30 20:18:14.073954 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.073417    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" containerID="cri-o://8aba076b8f671e9a06c675594b74e6408b1883c4c7d6e9cc157bbb4592659544" gracePeriod=45
Oct 30 20:18:14.078099 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.078006    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": read tcp 10.133.0.2:36914->10.133.0.53:8000: read: connection reset by peer"
Oct 30 20:18:14.091181 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.091151    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" containerID="cri-o://042978397fd8ec66811157de161746daa23ab0563f53bdc5a109f693f9a557ff" gracePeriod=45
Oct 30 20:18:14.102314 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.102282    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main" containerID="cri-o://f25e78f30d5324cbc95da380ecf11defe928c936e3d65e412adc84d099756d59" gracePeriod=45
Oct 30 20:18:14.125916 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.125890    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 30 20:18:14.190571 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.190534    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": dial tcp 10.133.0.39:8000: connect: connection refused"
Oct 30 20:18:14.386615 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.386585    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 20:18:14.414578 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.409337    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 20:18:14.421664 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.421636    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 20:18:14.444404 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.444375    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 20:18:14.499536 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.499491    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-home\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-home\") pod \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") "
Oct 30 20:18:14.499725 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.499541    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f531fa87-3d67-4d59-8c6e-9869a8852459-tls-certs\" (UniqueName: \"kubernetes.io/secret/f531fa87-3d67-4d59-8c6e-9869a8852459-tls-certs\") pod \"f531fa87-3d67-4d59-8c6e-9869a8852459\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") "
Oct 30 20:18:14.499725 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.499574    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-model-cache\") pod \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") "
Oct 30 20:18:14.499725 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.499630    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-home\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-home\") pod \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") "
Oct 30 20:18:14.499725 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.499680    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-model-cache\") pod \"37429652-9f2c-4abc-b27a-598fad5ab338\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") "
Oct 30 20:18:14.499725 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.499707    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37429652-9f2c-4abc-b27a-598fad5ab338-tls-certs\" (UniqueName: \"kubernetes.io/secret/37429652-9f2c-4abc-b27a-598fad5ab338-tls-certs\") pod \"37429652-9f2c-4abc-b27a-598fad5ab338\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") "
Oct 30 20:18:14.500025 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.499740    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-tmp-dir\") pod \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") "
Oct 30 20:18:14.500025 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.499766    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-tls-certs\") pod \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") "
Oct 30 20:18:14.500025 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.499776    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-home" pod "0772e44f-03d9-4cf5-a642-c0e0a50b27e9" (UID: "0772e44f-03d9-4cf5-a642-c0e0a50b27e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:14.500025 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.499793    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-home\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-home\") pod \"f531fa87-3d67-4d59-8c6e-9869a8852459\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") "
Oct 30 20:18:14.500025 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.499871    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-dshm\") pod \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") "
Oct 30 20:18:14.500025 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.499900    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-dshm\") pod \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") "
Oct 30 20:18:14.500025 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.499966    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-dshm\") pod \"37429652-9f2c-4abc-b27a-598fad5ab338\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") "
Oct 30 20:18:14.500025 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.500008    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-model-cache" pod "37429652-9f2c-4abc-b27a-598fad5ab338" (UID: "37429652-9f2c-4abc-b27a-598fad5ab338"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:14.500550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.500037    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-tmp-dir\") pod \"f531fa87-3d67-4d59-8c6e-9869a8852459\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") "
Oct 30 20:18:14.500550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.500217    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-model-cache" pod "2d6f9a7a-40c2-4169-b82a-48cd170dd8db" (UID: "2d6f9a7a-40c2-4169-b82a-48cd170dd8db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:14.500550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.500231    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37429652-9f2c-4abc-b27a-598fad5ab338-kube-api-access-n24k9\" (UniqueName: \"kubernetes.io/projected/37429652-9f2c-4abc-b27a-598fad5ab338-kube-api-access-n24k9\") pod \"37429652-9f2c-4abc-b27a-598fad5ab338\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") "
Oct 30 20:18:14.500550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.500308    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-home\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-home\") pod \"37429652-9f2c-4abc-b27a-598fad5ab338\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") "
Oct 30 20:18:14.500550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.500348    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-tmp-dir\") pod \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") "
Oct 30 20:18:14.500550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.500374    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-tls-certs\") pod \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") "
Oct 30 20:18:14.500550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.500426    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-model-cache\") pod \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") "
Oct 30 20:18:14.500550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.500461    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-dshm\") pod \"f531fa87-3d67-4d59-8c6e-9869a8852459\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") "
Oct 30 20:18:14.500550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.500486    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-tmp-dir\") pod \"37429652-9f2c-4abc-b27a-598fad5ab338\" (UID: \"37429652-9f2c-4abc-b27a-598fad5ab338\") "
Oct 30 20:18:14.500550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.500534    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-model-cache\") pod \"f531fa87-3d67-4d59-8c6e-9869a8852459\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") "
Oct 30 20:18:14.501347 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.500572    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f531fa87-3d67-4d59-8c6e-9869a8852459-kube-api-access-4pxs7\" (UniqueName: \"kubernetes.io/projected/f531fa87-3d67-4d59-8c6e-9869a8852459-kube-api-access-4pxs7\") pod \"f531fa87-3d67-4d59-8c6e-9869a8852459\" (UID: \"f531fa87-3d67-4d59-8c6e-9869a8852459\") "
Oct 30 20:18:14.501347 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.500616    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-kube-api-access-nfrl4\" (UniqueName: \"kubernetes.io/projected/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-kube-api-access-nfrl4\") pod \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\" (UID: \"2d6f9a7a-40c2-4169-b82a-48cd170dd8db\") "
Oct 30 20:18:14.501347 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.500665    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-kube-api-access-57wc7\" (UniqueName: \"kubernetes.io/projected/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-kube-api-access-57wc7\") pod \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\" (UID: \"0772e44f-03d9-4cf5-a642-c0e0a50b27e9\") "
Oct 30 20:18:14.501347 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.501085    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-tmp-dir" pod "0772e44f-03d9-4cf5-a642-c0e0a50b27e9" (UID: "0772e44f-03d9-4cf5-a642-c0e0a50b27e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:14.501571 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.501409    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-home" pod "2d6f9a7a-40c2-4169-b82a-48cd170dd8db" (UID: "2d6f9a7a-40c2-4169-b82a-48cd170dd8db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:14.501571 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.501441    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.501571 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.501464    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.501571 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.501479    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.501571 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.501495    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.502208 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.502178    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-home" pod "f531fa87-3d67-4d59-8c6e-9869a8852459" (UID: "f531fa87-3d67-4d59-8c6e-9869a8852459"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:14.502586 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.502391    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-tmp-dir" pod "37429652-9f2c-4abc-b27a-598fad5ab338" (UID: "37429652-9f2c-4abc-b27a-598fad5ab338"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:14.502675 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.502631    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-model-cache" pod "0772e44f-03d9-4cf5-a642-c0e0a50b27e9" (UID: "0772e44f-03d9-4cf5-a642-c0e0a50b27e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:14.503167 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.502893    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-tmp-dir" pod "f531fa87-3d67-4d59-8c6e-9869a8852459" (UID: "f531fa87-3d67-4d59-8c6e-9869a8852459"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:14.503167 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.503146    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-model-cache" pod "f531fa87-3d67-4d59-8c6e-9869a8852459" (UID: "f531fa87-3d67-4d59-8c6e-9869a8852459"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:14.504647 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.504599    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-dshm" pod "0772e44f-03d9-4cf5-a642-c0e0a50b27e9" (UID: "0772e44f-03d9-4cf5-a642-c0e0a50b27e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:14.504778 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.504738    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-home" pod "37429652-9f2c-4abc-b27a-598fad5ab338" (UID: "37429652-9f2c-4abc-b27a-598fad5ab338"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:14.504853 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.504802    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-tmp-dir" pod "2d6f9a7a-40c2-4169-b82a-48cd170dd8db" (UID: "2d6f9a7a-40c2-4169-b82a-48cd170dd8db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:14.506384 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.506343    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-dshm" pod "37429652-9f2c-4abc-b27a-598fad5ab338" (UID: "37429652-9f2c-4abc-b27a-598fad5ab338"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:14.508850 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.508816    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f531fa87-3d67-4d59-8c6e-9869a8852459-kube-api-access-4pxs7" pod "f531fa87-3d67-4d59-8c6e-9869a8852459" (UID: "f531fa87-3d67-4d59-8c6e-9869a8852459"). InnerVolumeSpecName "kube-api-access-4pxs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:14.508959 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.508934    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-kube-api-access-nfrl4" pod "2d6f9a7a-40c2-4169-b82a-48cd170dd8db" (UID: "2d6f9a7a-40c2-4169-b82a-48cd170dd8db"). InnerVolumeSpecName "kube-api-access-nfrl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:14.509023 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.509000    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-dshm" pod "f531fa87-3d67-4d59-8c6e-9869a8852459" (UID: "f531fa87-3d67-4d59-8c6e-9869a8852459"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:14.509182 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.509161    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-tls-certs" pod "2d6f9a7a-40c2-4169-b82a-48cd170dd8db" (UID: "2d6f9a7a-40c2-4169-b82a-48cd170dd8db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:14.509182 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.509169    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-kube-api-access-57wc7" pod "0772e44f-03d9-4cf5-a642-c0e0a50b27e9" (UID: "0772e44f-03d9-4cf5-a642-c0e0a50b27e9"). InnerVolumeSpecName "kube-api-access-57wc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:14.509348 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.509212    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-tls-certs" pod "0772e44f-03d9-4cf5-a642-c0e0a50b27e9" (UID: "0772e44f-03d9-4cf5-a642-c0e0a50b27e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:14.509348 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.509297    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37429652-9f2c-4abc-b27a-598fad5ab338-tls-certs" pod "37429652-9f2c-4abc-b27a-598fad5ab338" (UID: "37429652-9f2c-4abc-b27a-598fad5ab338"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:14.509498 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.509358    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f531fa87-3d67-4d59-8c6e-9869a8852459-tls-certs" pod "f531fa87-3d67-4d59-8c6e-9869a8852459" (UID: "f531fa87-3d67-4d59-8c6e-9869a8852459"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:14.509552 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.509532    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-dshm" pod "2d6f9a7a-40c2-4169-b82a-48cd170dd8db" (UID: "2d6f9a7a-40c2-4169-b82a-48cd170dd8db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:14.510691 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.510671    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37429652-9f2c-4abc-b27a-598fad5ab338-kube-api-access-n24k9" pod "37429652-9f2c-4abc-b27a-598fad5ab338" (UID: "37429652-9f2c-4abc-b27a-598fad5ab338"). InnerVolumeSpecName "kube-api-access-n24k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:14.602855 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602777    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57wc7\" (UniqueName: \"kubernetes.io/projected/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-kube-api-access-57wc7\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.602855 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602809    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f531fa87-3d67-4d59-8c6e-9869a8852459-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.602855 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602820    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.602855 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602829    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37429652-9f2c-4abc-b27a-598fad5ab338-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.602855 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602837    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.602855 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602845    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.602855 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602852    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.602855 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602860    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.603270 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602867    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.603270 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602874    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.603270 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602882    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n24k9\" (UniqueName: \"kubernetes.io/projected/37429652-9f2c-4abc-b27a-598fad5ab338-kube-api-access-n24k9\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.603270 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602890    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.603270 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602899    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.603270 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602906    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.603270 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602914    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0772e44f-03d9-4cf5-a642-c0e0a50b27e9-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.603270 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602922    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.603270 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602930    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37429652-9f2c-4abc-b27a-598fad5ab338-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.603270 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602937    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f531fa87-3d67-4d59-8c6e-9869a8852459-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.603270 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602945    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pxs7\" (UniqueName: \"kubernetes.io/projected/f531fa87-3d67-4d59-8c6e-9869a8852459-kube-api-access-4pxs7\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.603270 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.602953    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfrl4\" (UniqueName: \"kubernetes.io/projected/2d6f9a7a-40c2-4169-b82a-48cd170dd8db-kube-api-access-nfrl4\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:14.840831 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.840792    6030 generic.go:356] "Generic (PLEG): container finished" podID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerID="8aba076b8f671e9a06c675594b74e6408b1883c4c7d6e9cc157bbb4592659544" exitCode=0
Oct 30 20:18:14.841008 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.840834    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"
Oct 30 20:18:14.841008 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.840887    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" event={"ID":"0772e44f-03d9-4cf5-a642-c0e0a50b27e9","Type":"ContainerDied","Data":"8aba076b8f671e9a06c675594b74e6408b1883c4c7d6e9cc157bbb4592659544"}
Oct 30 20:18:14.841008 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.840927    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw" event={"ID":"0772e44f-03d9-4cf5-a642-c0e0a50b27e9","Type":"ContainerDied","Data":"2c15749b45e223e3b44f905bd8de201551f15ba493dfbbdbe93b0a26699d512b"}
Oct 30 20:18:14.841008 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.840947    6030 scope.go:122] "RemoveContainer" containerID="8aba076b8f671e9a06c675594b74e6408b1883c4c7d6e9cc157bbb4592659544"
Oct 30 20:18:14.842822 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.842797    6030 generic.go:356] "Generic (PLEG): container finished" podID="37429652-9f2c-4abc-b27a-598fad5ab338" containerID="8cfb8bc3d11d8b4c7a8b51f3b4b68a13b44f47872dd11af4274a347e16b5a20f" exitCode=0
Oct 30 20:18:14.842938 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.842841    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"
Oct 30 20:18:14.842938 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.842887    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" event={"ID":"37429652-9f2c-4abc-b27a-598fad5ab338","Type":"ContainerDied","Data":"8cfb8bc3d11d8b4c7a8b51f3b4b68a13b44f47872dd11af4274a347e16b5a20f"}
Oct 30 20:18:14.843164 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.842935    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6" event={"ID":"37429652-9f2c-4abc-b27a-598fad5ab338","Type":"ContainerDied","Data":"e2ac84ef03d853b0216faf35d31ebdee6d51974787e09dc6d68ee6c68a7589bd"}
Oct 30 20:18:14.844544 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.844519    6030 generic.go:356] "Generic (PLEG): container finished" podID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerID="f25e78f30d5324cbc95da380ecf11defe928c936e3d65e412adc84d099756d59" exitCode=0
Oct 30 20:18:14.844765 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.844585    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"
Oct 30 20:18:14.844879 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.844600    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" event={"ID":"2d6f9a7a-40c2-4169-b82a-48cd170dd8db","Type":"ContainerDied","Data":"f25e78f30d5324cbc95da380ecf11defe928c936e3d65e412adc84d099756d59"}
Oct 30 20:18:14.844943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.844874    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj" event={"ID":"2d6f9a7a-40c2-4169-b82a-48cd170dd8db","Type":"ContainerDied","Data":"1163cc1337e75daf7ee1fea333498db7cbcb29c2c6c266db6e61f3fcd159792e"}
Oct 30 20:18:14.846678 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.846654    6030 generic.go:356] "Generic (PLEG): container finished" podID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerID="042978397fd8ec66811157de161746daa23ab0563f53bdc5a109f693f9a557ff" exitCode=0
Oct 30 20:18:14.846787 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.846683    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"
Oct 30 20:18:14.846787 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.846692    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" event={"ID":"f531fa87-3d67-4d59-8c6e-9869a8852459","Type":"ContainerDied","Data":"042978397fd8ec66811157de161746daa23ab0563f53bdc5a109f693f9a557ff"}
Oct 30 20:18:14.846787 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.846722    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67" event={"ID":"f531fa87-3d67-4d59-8c6e-9869a8852459","Type":"ContainerDied","Data":"9cba54c5c77a7ce6e3171ed8d94d4a3b66d579069172817db769bb224f4d8bc9"}
Oct 30 20:18:14.852635 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.852613    6030 scope.go:122] "RemoveContainer" containerID="8aba076b8f671e9a06c675594b74e6408b1883c4c7d6e9cc157bbb4592659544"
Oct 30 20:18:14.852970 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:18:14.852916    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aba076b8f671e9a06c675594b74e6408b1883c4c7d6e9cc157bbb4592659544\": container with ID starting with 8aba076b8f671e9a06c675594b74e6408b1883c4c7d6e9cc157bbb4592659544 not found: ID does not exist" containerID="8aba076b8f671e9a06c675594b74e6408b1883c4c7d6e9cc157bbb4592659544"
Oct 30 20:18:14.852970 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.852947    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aba076b8f671e9a06c675594b74e6408b1883c4c7d6e9cc157bbb4592659544"} err="failed to get container status \"8aba076b8f671e9a06c675594b74e6408b1883c4c7d6e9cc157bbb4592659544\": rpc error: code = NotFound desc = could not find container \"8aba076b8f671e9a06c675594b74e6408b1883c4c7d6e9cc157bbb4592659544\": container with ID starting with 8aba076b8f671e9a06c675594b74e6408b1883c4c7d6e9cc157bbb4592659544 not found: ID does not exist"
Oct 30 20:18:14.852970 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.852964    6030 scope.go:122] "RemoveContainer" containerID="8cfb8bc3d11d8b4c7a8b51f3b4b68a13b44f47872dd11af4274a347e16b5a20f"
Oct 30 20:18:14.866238 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.866221    6030 scope.go:122] "RemoveContainer" containerID="8cfb8bc3d11d8b4c7a8b51f3b4b68a13b44f47872dd11af4274a347e16b5a20f"
Oct 30 20:18:14.866564 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:18:14.866534    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cfb8bc3d11d8b4c7a8b51f3b4b68a13b44f47872dd11af4274a347e16b5a20f\": container with ID starting with 8cfb8bc3d11d8b4c7a8b51f3b4b68a13b44f47872dd11af4274a347e16b5a20f not found: ID does not exist" containerID="8cfb8bc3d11d8b4c7a8b51f3b4b68a13b44f47872dd11af4274a347e16b5a20f"
Oct 30 20:18:14.866615 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.866577    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cfb8bc3d11d8b4c7a8b51f3b4b68a13b44f47872dd11af4274a347e16b5a20f"} err="failed to get container status \"8cfb8bc3d11d8b4c7a8b51f3b4b68a13b44f47872dd11af4274a347e16b5a20f\": rpc error: code = NotFound desc = could not find container \"8cfb8bc3d11d8b4c7a8b51f3b4b68a13b44f47872dd11af4274a347e16b5a20f\": container with ID starting with 8cfb8bc3d11d8b4c7a8b51f3b4b68a13b44f47872dd11af4274a347e16b5a20f not found: ID does not exist"
Oct 30 20:18:14.866615 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.866601    6030 scope.go:122] "RemoveContainer" containerID="f25e78f30d5324cbc95da380ecf11defe928c936e3d65e412adc84d099756d59"
Oct 30 20:18:14.875155 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.875132    6030 scope.go:122] "RemoveContainer" containerID="f25e78f30d5324cbc95da380ecf11defe928c936e3d65e412adc84d099756d59"
Oct 30 20:18:14.875440 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:18:14.875422    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f25e78f30d5324cbc95da380ecf11defe928c936e3d65e412adc84d099756d59\": container with ID starting with f25e78f30d5324cbc95da380ecf11defe928c936e3d65e412adc84d099756d59 not found: ID does not exist" containerID="f25e78f30d5324cbc95da380ecf11defe928c936e3d65e412adc84d099756d59"
Oct 30 20:18:14.875489 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.875447    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f25e78f30d5324cbc95da380ecf11defe928c936e3d65e412adc84d099756d59"} err="failed to get container status \"f25e78f30d5324cbc95da380ecf11defe928c936e3d65e412adc84d099756d59\": rpc error: code = NotFound desc = could not find container \"f25e78f30d5324cbc95da380ecf11defe928c936e3d65e412adc84d099756d59\": container with ID starting with f25e78f30d5324cbc95da380ecf11defe928c936e3d65e412adc84d099756d59 not found: ID does not exist"
Oct 30 20:18:14.875489 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.875462    6030 scope.go:122] "RemoveContainer" containerID="042978397fd8ec66811157de161746daa23ab0563f53bdc5a109f693f9a557ff"
Oct 30 20:18:14.881076 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.881030    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"]
Oct 30 20:18:14.885547 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.885518    6030 scope.go:122] "RemoveContainer" containerID="042978397fd8ec66811157de161746daa23ab0563f53bdc5a109f693f9a557ff"
Oct 30 20:18:14.886374 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:18:14.886350    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042978397fd8ec66811157de161746daa23ab0563f53bdc5a109f693f9a557ff\": container with ID starting with 042978397fd8ec66811157de161746daa23ab0563f53bdc5a109f693f9a557ff not found: ID does not exist" containerID="042978397fd8ec66811157de161746daa23ab0563f53bdc5a109f693f9a557ff"
Oct 30 20:18:14.886458 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.886383    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042978397fd8ec66811157de161746daa23ab0563f53bdc5a109f693f9a557ff"} err="failed to get container status \"042978397fd8ec66811157de161746daa23ab0563f53bdc5a109f693f9a557ff\": rpc error: code = NotFound desc = could not find container \"042978397fd8ec66811157de161746daa23ab0563f53bdc5a109f693f9a557ff\": container with ID starting with 042978397fd8ec66811157de161746daa23ab0563f53bdc5a109f693f9a557ff not found: ID does not exist"
Oct 30 20:18:14.888717 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.888699    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/simulator-w2-f06af6-kserve-56c56c6d5b-f29fj"]
Oct 30 20:18:14.901314 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.901293    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"]
Oct 30 20:18:14.906397 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.906374    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/distinct-w2-f06af6-kserve-6d647f857b-xgpmw"]
Oct 30 20:18:14.920225 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.920201    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"]
Oct 30 20:18:14.923196 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.923175    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/distinct-2-w2-f06af6-kserve-85cc78bfcf-xznm6"]
Oct 30 20:18:14.952193 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.952170    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"]
Oct 30 20:18:15.006069 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:14.994202    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f06af6/premium-w2-f06af6-kserve-64d7d659c-gkw67"]
Oct 30 20:18:15.206132 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:15.206101    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" path="/var/lib/kubelet/pods/0772e44f-03d9-4cf5-a642-c0e0a50b27e9/volumes"
Oct 30 20:18:15.206618 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:15.206602    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" path="/var/lib/kubelet/pods/2d6f9a7a-40c2-4169-b82a-48cd170dd8db/volumes"
Oct 30 20:18:15.207011 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:15.206997    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" path="/var/lib/kubelet/pods/37429652-9f2c-4abc-b27a-598fad5ab338/volumes"
Oct 30 20:18:15.208005 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:15.207732    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" path="/var/lib/kubelet/pods/f531fa87-3d67-4d59-8c6e-9869a8852459/volumes"
Oct 30 20:18:17.047777 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.047742    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main" containerID="cri-o://f430a587d812c0287f1396b41c53274c8994242b4ca6c67fdf7c80fd9125a46e" gracePeriod=45
Oct 30 20:18:17.061080 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.061021    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main" containerID="cri-o://6b3bf745e06bad98938d27f2da3f22a5ca67bce3dfa85230385c9b98c9e3ae59" gracePeriod=45
Oct 30 20:18:17.365552 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.365531    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 20:18:17.368718 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.368697    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 20:18:17.439443 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.439406    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-tmp-dir\") pod \"e7f34b33-c655-42b7-866f-6d61375e0f65\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") "
Oct 30 20:18:17.439619 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.439470    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-dshm\") pod \"e7f34b33-c655-42b7-866f-6d61375e0f65\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") "
Oct 30 20:18:17.439619 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.439503    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-model-cache\") pod \"b88e4980-4558-4da6-98a5-d59a6fb264eb\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") "
Oct 30 20:18:17.439619 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.439574    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f34b33-c655-42b7-866f-6d61375e0f65-kube-api-access-rrkrr\" (UniqueName: \"kubernetes.io/projected/e7f34b33-c655-42b7-866f-6d61375e0f65-kube-api-access-rrkrr\") pod \"e7f34b33-c655-42b7-866f-6d61375e0f65\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") "
Oct 30 20:18:17.439619 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.439608    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b88e4980-4558-4da6-98a5-d59a6fb264eb-tls-certs\" (UniqueName: \"kubernetes.io/secret/b88e4980-4558-4da6-98a5-d59a6fb264eb-tls-certs\") pod \"b88e4980-4558-4da6-98a5-d59a6fb264eb\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") "
Oct 30 20:18:17.439871 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.439667    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-tmp-dir\") pod \"b88e4980-4558-4da6-98a5-d59a6fb264eb\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") "
Oct 30 20:18:17.439871 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.439728    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-dshm\") pod \"b88e4980-4558-4da6-98a5-d59a6fb264eb\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") "
Oct 30 20:18:17.439871 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.439753    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f34b33-c655-42b7-866f-6d61375e0f65-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f34b33-c655-42b7-866f-6d61375e0f65-tls-certs\") pod \"e7f34b33-c655-42b7-866f-6d61375e0f65\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") "
Oct 30 20:18:17.439871 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.439798    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-home\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-home\") pod \"b88e4980-4558-4da6-98a5-d59a6fb264eb\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") "
Oct 30 20:18:17.439871 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.439840    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-model-cache\") pod \"e7f34b33-c655-42b7-866f-6d61375e0f65\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") "
Oct 30 20:18:17.440194 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.439894    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-model-cache" pod "b88e4980-4558-4da6-98a5-d59a6fb264eb" (UID: "b88e4980-4558-4da6-98a5-d59a6fb264eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:17.440194 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.439905    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-home\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-home\") pod \"e7f34b33-c655-42b7-866f-6d61375e0f65\" (UID: \"e7f34b33-c655-42b7-866f-6d61375e0f65\") "
Oct 30 20:18:17.440194 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.440017    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88e4980-4558-4da6-98a5-d59a6fb264eb-kube-api-access-stqr6\" (UniqueName: \"kubernetes.io/projected/b88e4980-4558-4da6-98a5-d59a6fb264eb-kube-api-access-stqr6\") pod \"b88e4980-4558-4da6-98a5-d59a6fb264eb\" (UID: \"b88e4980-4558-4da6-98a5-d59a6fb264eb\") "
Oct 30 20:18:17.440194 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.440131    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-home" pod "e7f34b33-c655-42b7-866f-6d61375e0f65" (UID: "e7f34b33-c655-42b7-866f-6d61375e0f65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:17.440655 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.440628    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:17.440780 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.440660    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:17.441008 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.440983    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-tmp-dir" pod "e7f34b33-c655-42b7-866f-6d61375e0f65" (UID: "e7f34b33-c655-42b7-866f-6d61375e0f65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:17.441471 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.441441    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-home" pod "b88e4980-4558-4da6-98a5-d59a6fb264eb" (UID: "b88e4980-4558-4da6-98a5-d59a6fb264eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:17.441670 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.441646    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-model-cache" pod "e7f34b33-c655-42b7-866f-6d61375e0f65" (UID: "e7f34b33-c655-42b7-866f-6d61375e0f65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:17.444618 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.444248    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-tmp-dir" pod "b88e4980-4558-4da6-98a5-d59a6fb264eb" (UID: "b88e4980-4558-4da6-98a5-d59a6fb264eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:17.452875 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.452848    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f34b33-c655-42b7-866f-6d61375e0f65-tls-certs" pod "e7f34b33-c655-42b7-866f-6d61375e0f65" (UID: "e7f34b33-c655-42b7-866f-6d61375e0f65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:17.455458 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.455432    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-dshm" pod "b88e4980-4558-4da6-98a5-d59a6fb264eb" (UID: "b88e4980-4558-4da6-98a5-d59a6fb264eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:17.455586 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.455551    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88e4980-4558-4da6-98a5-d59a6fb264eb-kube-api-access-stqr6" pod "b88e4980-4558-4da6-98a5-d59a6fb264eb" (UID: "b88e4980-4558-4da6-98a5-d59a6fb264eb"). InnerVolumeSpecName "kube-api-access-stqr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:17.455705 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.455686    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-dshm" pod "e7f34b33-c655-42b7-866f-6d61375e0f65" (UID: "e7f34b33-c655-42b7-866f-6d61375e0f65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:17.457823 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.456820    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f34b33-c655-42b7-866f-6d61375e0f65-kube-api-access-rrkrr" pod "e7f34b33-c655-42b7-866f-6d61375e0f65" (UID: "e7f34b33-c655-42b7-866f-6d61375e0f65"). InnerVolumeSpecName "kube-api-access-rrkrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:17.457823 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.457666    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b88e4980-4558-4da6-98a5-d59a6fb264eb-tls-certs" pod "b88e4980-4558-4da6-98a5-d59a6fb264eb" (UID: "b88e4980-4558-4da6-98a5-d59a6fb264eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:17.541558 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.541524    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stqr6\" (UniqueName: \"kubernetes.io/projected/b88e4980-4558-4da6-98a5-d59a6fb264eb-kube-api-access-stqr6\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:17.541558 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.541556    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:17.541558 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.541567    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:17.541786 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.541575    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrkrr\" (UniqueName: \"kubernetes.io/projected/e7f34b33-c655-42b7-866f-6d61375e0f65-kube-api-access-rrkrr\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:17.541786 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.541585    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b88e4980-4558-4da6-98a5-d59a6fb264eb-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:17.541786 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.541593    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:17.541786 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.541601    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:17.541786 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.541609    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f34b33-c655-42b7-866f-6d61375e0f65-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:17.541786 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.541616    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b88e4980-4558-4da6-98a5-d59a6fb264eb-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:17.541786 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.541624    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f34b33-c655-42b7-866f-6d61375e0f65-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:17.869943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.869903    6030 generic.go:356] "Generic (PLEG): container finished" podID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerID="6b3bf745e06bad98938d27f2da3f22a5ca67bce3dfa85230385c9b98c9e3ae59" exitCode=0
Oct 30 20:18:17.870211 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.869962    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"
Oct 30 20:18:17.870211 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.869997    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" event={"ID":"b88e4980-4558-4da6-98a5-d59a6fb264eb","Type":"ContainerDied","Data":"6b3bf745e06bad98938d27f2da3f22a5ca67bce3dfa85230385c9b98c9e3ae59"}
Oct 30 20:18:17.870211 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.870058    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws" event={"ID":"b88e4980-4558-4da6-98a5-d59a6fb264eb","Type":"ContainerDied","Data":"8e48add4cbe3520c0a4670b1e3e474c460195216062bbf2a3ac89a19f9be4acd"}
Oct 30 20:18:17.870211 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.870074    6030 scope.go:122] "RemoveContainer" containerID="6b3bf745e06bad98938d27f2da3f22a5ca67bce3dfa85230385c9b98c9e3ae59"
Oct 30 20:18:17.871620 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.871594    6030 generic.go:356] "Generic (PLEG): container finished" podID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerID="f430a587d812c0287f1396b41c53274c8994242b4ca6c67fdf7c80fd9125a46e" exitCode=0
Oct 30 20:18:17.871737 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.871646    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"
Oct 30 20:18:17.871737 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.871658    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" event={"ID":"e7f34b33-c655-42b7-866f-6d61375e0f65","Type":"ContainerDied","Data":"f430a587d812c0287f1396b41c53274c8994242b4ca6c67fdf7c80fd9125a46e"}
Oct 30 20:18:17.871737 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.871688    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5" event={"ID":"e7f34b33-c655-42b7-866f-6d61375e0f65","Type":"ContainerDied","Data":"1e6c2da006852c471d8331d576cd8975b7e583e9e2345fbce2ba695f5edf7476"}
Oct 30 20:18:17.880613 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.880595    6030 scope.go:122] "RemoveContainer" containerID="6b3bf745e06bad98938d27f2da3f22a5ca67bce3dfa85230385c9b98c9e3ae59"
Oct 30 20:18:17.880866 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:18:17.880846    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3bf745e06bad98938d27f2da3f22a5ca67bce3dfa85230385c9b98c9e3ae59\": container with ID starting with 6b3bf745e06bad98938d27f2da3f22a5ca67bce3dfa85230385c9b98c9e3ae59 not found: ID does not exist" containerID="6b3bf745e06bad98938d27f2da3f22a5ca67bce3dfa85230385c9b98c9e3ae59"
Oct 30 20:18:17.880942 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.880877    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3bf745e06bad98938d27f2da3f22a5ca67bce3dfa85230385c9b98c9e3ae59"} err="failed to get container status \"6b3bf745e06bad98938d27f2da3f22a5ca67bce3dfa85230385c9b98c9e3ae59\": rpc error: code = NotFound desc = could not find container \"6b3bf745e06bad98938d27f2da3f22a5ca67bce3dfa85230385c9b98c9e3ae59\": container with ID starting with 6b3bf745e06bad98938d27f2da3f22a5ca67bce3dfa85230385c9b98c9e3ae59 not found: ID does not exist"
Oct 30 20:18:17.880942 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.880899    6030 scope.go:122] "RemoveContainer" containerID="f430a587d812c0287f1396b41c53274c8994242b4ca6c67fdf7c80fd9125a46e"
Oct 30 20:18:17.889461 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.889439    6030 scope.go:122] "RemoveContainer" containerID="f430a587d812c0287f1396b41c53274c8994242b4ca6c67fdf7c80fd9125a46e"
Oct 30 20:18:17.889692 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:18:17.889676    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f430a587d812c0287f1396b41c53274c8994242b4ca6c67fdf7c80fd9125a46e\": container with ID starting with f430a587d812c0287f1396b41c53274c8994242b4ca6c67fdf7c80fd9125a46e not found: ID does not exist" containerID="f430a587d812c0287f1396b41c53274c8994242b4ca6c67fdf7c80fd9125a46e"
Oct 30 20:18:17.889750 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.889700    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f430a587d812c0287f1396b41c53274c8994242b4ca6c67fdf7c80fd9125a46e"} err="failed to get container status \"f430a587d812c0287f1396b41c53274c8994242b4ca6c67fdf7c80fd9125a46e\": rpc error: code = NotFound desc = could not find container \"f430a587d812c0287f1396b41c53274c8994242b4ca6c67fdf7c80fd9125a46e\": container with ID starting with f430a587d812c0287f1396b41c53274c8994242b4ca6c67fdf7c80fd9125a46e not found: ID does not exist"
Oct 30 20:18:17.901424 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.901403    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"]
Oct 30 20:18:17.907960 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.907939    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/simulator-w3-c1fdd6-kserve-6874ffc766-vztws"]
Oct 30 20:18:17.921987 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.921964    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"]
Oct 30 20:18:17.926359 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:17.926341    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/distinct-w3-c1fdd6-kserve-74c74ddd6b-n9xq5"]
Oct 30 20:18:19.200751 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:19.200720    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" path="/var/lib/kubelet/pods/b88e4980-4558-4da6-98a5-d59a6fb264eb/volumes"
Oct 30 20:18:19.201297 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:19.201281    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" path="/var/lib/kubelet/pods/e7f34b33-c655-42b7-866f-6d61375e0f65/volumes"
Oct 30 20:18:26.195351 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.195197    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"]
Oct 30 20:18:26.195921 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.195896    6030 cpu_manager.go:414] "containerMap: removing container" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main"
Oct 30 20:18:26.196032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.195924    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main"
Oct 30 20:18:26.196032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.195955    6030 cpu_manager.go:414] "containerMap: removing container" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main"
Oct 30 20:18:26.196032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.195965    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main"
Oct 30 20:18:26.196032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.195981    6030 cpu_manager.go:414] "containerMap: removing container" podUID="40974280-ed25-41d5-8697-4c09d86aadb1" containerName="payload-pre-processing"
Oct 30 20:18:26.196032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.195990    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40974280-ed25-41d5-8697-4c09d86aadb1" containerName="payload-pre-processing"
Oct 30 20:18:26.196032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196006    6030 cpu_manager.go:414] "containerMap: removing container" podUID="c75bdfb0-89bd-4935-baae-55475fdb405a" containerName="istio-proxy"
Oct 30 20:18:26.196032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196011    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75bdfb0-89bd-4935-baae-55475fdb405a" containerName="istio-proxy"
Oct 30 20:18:26.196032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196019    6030 cpu_manager.go:414] "containerMap: removing container" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main"
Oct 30 20:18:26.196032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196023    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main"
Oct 30 20:18:26.196032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196037    6030 cpu_manager.go:414] "containerMap: removing container" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main"
Oct 30 20:18:26.196440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196078    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main"
Oct 30 20:18:26.196440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196088    6030 cpu_manager.go:414] "containerMap: removing container" podUID="1dfa9318-c8b7-4c06-91c5-f630b3dc7810" containerName="payload-processing"
Oct 30 20:18:26.196440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196092    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dfa9318-c8b7-4c06-91c5-f630b3dc7810" containerName="payload-processing"
Oct 30 20:18:26.196440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196107    6030 cpu_manager.go:414] "containerMap: removing container" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main"
Oct 30 20:18:26.196440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196111    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main"
Oct 30 20:18:26.196440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196129    6030 cpu_manager.go:414] "containerMap: removing container" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main"
Oct 30 20:18:26.196440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196133    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main"
Oct 30 20:18:26.196440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196220    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6f9a7a-40c2-4169-b82a-48cd170dd8db" containerName="main"
Oct 30 20:18:26.196440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196229    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dfa9318-c8b7-4c06-91c5-f630b3dc7810" containerName="payload-processing"
Oct 30 20:18:26.196440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196238    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88e4980-4558-4da6-98a5-d59a6fb264eb" containerName="main"
Oct 30 20:18:26.196440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196247    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="37429652-9f2c-4abc-b27a-598fad5ab338" containerName="main"
Oct 30 20:18:26.196440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196255    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f34b33-c655-42b7-866f-6d61375e0f65" containerName="main"
Oct 30 20:18:26.196440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196261    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75bdfb0-89bd-4935-baae-55475fdb405a" containerName="istio-proxy"
Oct 30 20:18:26.196440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196268    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="f531fa87-3d67-4d59-8c6e-9869a8852459" containerName="main"
Oct 30 20:18:26.196440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196275    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="40974280-ed25-41d5-8697-4c09d86aadb1" containerName="payload-pre-processing"
Oct 30 20:18:26.196440 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.196281    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="0772e44f-03d9-4cf5-a642-c0e0a50b27e9" containerName="main"
Oct 30 20:18:26.201431 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.201406    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"
Oct 30 20:18:26.212403 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.212117    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"]
Oct 30 20:18:26.212403 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.212361    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bdf9e6\""
Oct 30 20:18:26.213020 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.212978    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-bdf9e6-dockercfg-btwc5\""
Oct 30 20:18:26.254959 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.254805    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6lhb\" (UniqueName: \"kubernetes.io/projected/d67582dc-0db8-4943-836f-fe3d44c53065-kube-api-access-n6lhb\") pod \"payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j\" (UID: \"d67582dc-0db8-4943-836f-fe3d44c53065\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"
Oct 30 20:18:26.254959 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.254893    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67582dc-0db8-4943-836f-fe3d44c53065-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j\" (UID: \"d67582dc-0db8-4943-836f-fe3d44c53065\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"
Oct 30 20:18:26.365071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.359780    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6lhb\" (UniqueName: \"kubernetes.io/projected/d67582dc-0db8-4943-836f-fe3d44c53065-kube-api-access-n6lhb\") pod \"payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j\" (UID: \"d67582dc-0db8-4943-836f-fe3d44c53065\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"
Oct 30 20:18:26.365071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.359881    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67582dc-0db8-4943-836f-fe3d44c53065-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j\" (UID: \"d67582dc-0db8-4943-836f-fe3d44c53065\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"
Oct 30 20:18:26.365071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.360977    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67582dc-0db8-4943-836f-fe3d44c53065-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j\" (UID: \"d67582dc-0db8-4943-836f-fe3d44c53065\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"
Oct 30 20:18:26.396071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.395667    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6lhb\" (UniqueName: \"kubernetes.io/projected/d67582dc-0db8-4943-836f-fe3d44c53065-kube-api-access-n6lhb\") pod \"payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j\" (UID: \"d67582dc-0db8-4943-836f-fe3d44c53065\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"
Oct 30 20:18:26.521295 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.521215    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"
Oct 30 20:18:26.801444 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.801404    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"]
Oct 30 20:18:26.803138 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:18:26.803105    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67582dc_0db8_4943_836f_fe3d44c53065.slice/crio-b2a54e6ddc20dbed41a4c5d7564f69fad50d6dd378bf8e3c95cbff527cdc7724 WatchSource:0}: Error finding container b2a54e6ddc20dbed41a4c5d7564f69fad50d6dd378bf8e3c95cbff527cdc7724: Status 404 returned error can't find the container with id b2a54e6ddc20dbed41a4c5d7564f69fad50d6dd378bf8e3c95cbff527cdc7724
Oct 30 20:18:26.937785 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.937748    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j" event={"ID":"d67582dc-0db8-4943-836f-fe3d44c53065","Type":"ContainerStarted","Data":"191c17a20b495b6d4a786f7195081383ccb03bb62725a2055b383ad3eb8fa9e7"}
Oct 30 20:18:26.937959 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.937845    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"
Oct 30 20:18:26.937959 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.937869    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j" event={"ID":"d67582dc-0db8-4943-836f-fe3d44c53065","Type":"ContainerStarted","Data":"b2a54e6ddc20dbed41a4c5d7564f69fad50d6dd378bf8e3c95cbff527cdc7724"}
Oct 30 20:18:26.958987 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:26.958936    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j" podStartSLOduration=0.958922246 podStartE2EDuration="958.922246ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:26.95800788 +0000 UTC m=+2785.873614736" watchObservedRunningTime="2026-10-30 20:18:26.958922246 +0000 UTC m=+2785.874529098"
Oct 30 20:18:34.290759 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:18:34.290719    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:18:37.408078 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:37.408002    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"]
Oct 30 20:18:37.408583 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:37.408301    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5" podUID="be081d10-b3c7-4a5a-9abf-9989ca04e364" containerName="istio-proxy" containerID="cri-o://777297e7b845203657f6165e0e2eed83a5e4b838f0246bf2b2a02e687a5e2ae5" gracePeriod=30
Oct 30 20:18:37.947953 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:37.947924    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"
Oct 30 20:18:40.757007 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:40.756921    6030 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:18:40.757007 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:40.756985    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5" podUID="be081d10-b3c7-4a5a-9abf-9989ca04e364" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:18:42.673126 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.673100    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 20:18:42.804630 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.804549    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be081d10-b3c7-4a5a-9abf-9989ca04e364-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be081d10-b3c7-4a5a-9abf-9989ca04e364-istiod-ca-cert\") pod \"be081d10-b3c7-4a5a-9abf-9989ca04e364\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") "
Oct 30 20:18:42.804630 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.804591    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-podinfo\") pod \"be081d10-b3c7-4a5a-9abf-9989ca04e364\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") "
Oct 30 20:18:42.804630 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.804613    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-credential-socket\") pod \"be081d10-b3c7-4a5a-9abf-9989ca04e364\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") "
Oct 30 20:18:42.804954 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.804715    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-workload-certs\") pod \"be081d10-b3c7-4a5a-9abf-9989ca04e364\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") "
Oct 30 20:18:42.804954 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.804811    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be081d10-b3c7-4a5a-9abf-9989ca04e364-kube-api-access-zjxv6\" (UniqueName: \"kubernetes.io/projected/be081d10-b3c7-4a5a-9abf-9989ca04e364-kube-api-access-zjxv6\") pod \"be081d10-b3c7-4a5a-9abf-9989ca04e364\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") "
Oct 30 20:18:42.804954 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.804854    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-token\" (UniqueName: \"kubernetes.io/projected/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-token\") pod \"be081d10-b3c7-4a5a-9abf-9989ca04e364\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") "
Oct 30 20:18:42.804954 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.804858    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-credential-socket" pod "be081d10-b3c7-4a5a-9abf-9989ca04e364" (UID: "be081d10-b3c7-4a5a-9abf-9989ca04e364"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:42.805234 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.805022    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-data\") pod \"be081d10-b3c7-4a5a-9abf-9989ca04e364\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") "
Oct 30 20:18:42.805234 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.805079    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be081d10-b3c7-4a5a-9abf-9989ca04e364-istiod-ca-cert" pod "be081d10-b3c7-4a5a-9abf-9989ca04e364" (UID: "be081d10-b3c7-4a5a-9abf-9989ca04e364"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:42.805234 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.805088    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-workload-socket\") pod \"be081d10-b3c7-4a5a-9abf-9989ca04e364\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") "
Oct 30 20:18:42.805234 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.805165    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-envoy\") pod \"be081d10-b3c7-4a5a-9abf-9989ca04e364\" (UID: \"be081d10-b3c7-4a5a-9abf-9989ca04e364\") "
Oct 30 20:18:42.805475 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.805302    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-workload-certs" pod "be081d10-b3c7-4a5a-9abf-9989ca04e364" (UID: "be081d10-b3c7-4a5a-9abf-9989ca04e364"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:42.805475 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.805329    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-workload-socket" pod "be081d10-b3c7-4a5a-9abf-9989ca04e364" (UID: "be081d10-b3c7-4a5a-9abf-9989ca04e364"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:42.805704 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.805681    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-data" pod "be081d10-b3c7-4a5a-9abf-9989ca04e364" (UID: "be081d10-b3c7-4a5a-9abf-9989ca04e364"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:42.805891 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.805874    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be081d10-b3c7-4a5a-9abf-9989ca04e364-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:42.806005 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.805899    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:42.806005 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.805914    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:42.806005 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.805928    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:42.806005 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.805941    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:42.807939 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.807904    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-token" pod "be081d10-b3c7-4a5a-9abf-9989ca04e364" (UID: "be081d10-b3c7-4a5a-9abf-9989ca04e364"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:42.808068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.807933    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-podinfo" pod "be081d10-b3c7-4a5a-9abf-9989ca04e364" (UID: "be081d10-b3c7-4a5a-9abf-9989ca04e364"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:18:42.808141 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.808118    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be081d10-b3c7-4a5a-9abf-9989ca04e364-kube-api-access-zjxv6" pod "be081d10-b3c7-4a5a-9abf-9989ca04e364" (UID: "be081d10-b3c7-4a5a-9abf-9989ca04e364"). InnerVolumeSpecName "kube-api-access-zjxv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:42.808443 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.808420    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-envoy" pod "be081d10-b3c7-4a5a-9abf-9989ca04e364" (UID: "be081d10-b3c7-4a5a-9abf-9989ca04e364"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:42.900479 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.900442    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"]
Oct 30 20:18:42.900720 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.900693    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t" podUID="ccdec0db-9dd5-4885-b6e0-0583623ad967" containerName="istio-proxy" containerID="cri-o://7dd902950fb1647ef0caba07bdeff0b410ba9cd97a3a12e28ceb36328f767d5d" gracePeriod=30
Oct 30 20:18:42.907703 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.907656    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:42.907703 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.907692    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjxv6\" (UniqueName: \"kubernetes.io/projected/be081d10-b3c7-4a5a-9abf-9989ca04e364-kube-api-access-zjxv6\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:42.908091 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.907708    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:42.908091 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:42.907722    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be081d10-b3c7-4a5a-9abf-9989ca04e364-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:43.047113 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:43.047074    6030 generic.go:356] "Generic (PLEG): container finished" podID="be081d10-b3c7-4a5a-9abf-9989ca04e364" containerID="777297e7b845203657f6165e0e2eed83a5e4b838f0246bf2b2a02e687a5e2ae5" exitCode=0
Oct 30 20:18:43.047312 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:43.047143    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"
Oct 30 20:18:43.047312 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:43.047142    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5" event={"ID":"be081d10-b3c7-4a5a-9abf-9989ca04e364","Type":"ContainerDied","Data":"777297e7b845203657f6165e0e2eed83a5e4b838f0246bf2b2a02e687a5e2ae5"}
Oct 30 20:18:43.047312 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:43.047243    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5" event={"ID":"be081d10-b3c7-4a5a-9abf-9989ca04e364","Type":"ContainerDied","Data":"74c23fdb7d941a5f59023aa2250b9e176760f15a9dfffab64d21d7dd923b582e"}
Oct 30 20:18:43.047312 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:43.047275    6030 scope.go:122] "RemoveContainer" containerID="777297e7b845203657f6165e0e2eed83a5e4b838f0246bf2b2a02e687a5e2ae5"
Oct 30 20:18:43.067273 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:43.067250    6030 scope.go:122] "RemoveContainer" containerID="777297e7b845203657f6165e0e2eed83a5e4b838f0246bf2b2a02e687a5e2ae5"
Oct 30 20:18:43.067646 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:18:43.067620    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777297e7b845203657f6165e0e2eed83a5e4b838f0246bf2b2a02e687a5e2ae5\": container with ID starting with 777297e7b845203657f6165e0e2eed83a5e4b838f0246bf2b2a02e687a5e2ae5 not found: ID does not exist" containerID="777297e7b845203657f6165e0e2eed83a5e4b838f0246bf2b2a02e687a5e2ae5"
Oct 30 20:18:43.067771 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:43.067658    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777297e7b845203657f6165e0e2eed83a5e4b838f0246bf2b2a02e687a5e2ae5"} err="failed to get container status \"777297e7b845203657f6165e0e2eed83a5e4b838f0246bf2b2a02e687a5e2ae5\": rpc error: code = NotFound desc = could not find container \"777297e7b845203657f6165e0e2eed83a5e4b838f0246bf2b2a02e687a5e2ae5\": container with ID starting with 777297e7b845203657f6165e0e2eed83a5e4b838f0246bf2b2a02e687a5e2ae5 not found: ID does not exist"
Oct 30 20:18:43.090987 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:43.090933    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"]
Oct 30 20:18:43.097326 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:43.097289    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-tp4x5"]
Oct 30 20:18:43.202308 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:43.202279    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be081d10-b3c7-4a5a-9abf-9989ca04e364" path="/var/lib/kubelet/pods/be081d10-b3c7-4a5a-9abf-9989ca04e364/volumes"
Oct 30 20:18:48.096115 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.096084    6030 generic.go:356] "Generic (PLEG): container finished" podID="ccdec0db-9dd5-4885-b6e0-0583623ad967" containerID="7dd902950fb1647ef0caba07bdeff0b410ba9cd97a3a12e28ceb36328f767d5d" exitCode=0
Oct 30 20:18:48.096816 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.096792    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t" event={"ID":"ccdec0db-9dd5-4885-b6e0-0583623ad967","Type":"ContainerDied","Data":"7dd902950fb1647ef0caba07bdeff0b410ba9cd97a3a12e28ceb36328f767d5d"}
Oct 30 20:18:48.097122 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.097101    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t" event={"ID":"ccdec0db-9dd5-4885-b6e0-0583623ad967","Type":"ContainerDied","Data":"93cca299a16cdd25f993377c5905887a806e5e7a04e732319d279adaa47807fc"}
Oct 30 20:18:48.097264 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.097250    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93cca299a16cdd25f993377c5905887a806e5e7a04e732319d279adaa47807fc"
Oct 30 20:18:48.097419 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.097409    6030 scope.go:122] "RemoveContainer" containerID="7a2baa1c2bbf44b9136bc0f6aea8da47da956ec360cf61a49f0d6e454b9f9efd"
Oct 30 20:18:48.098879 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.098510    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 20:18:48.185184 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.184413    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-workload-socket\") pod \"ccdec0db-9dd5-4885-b6e0-0583623ad967\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") "
Oct 30 20:18:48.185184 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.184494    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccdec0db-9dd5-4885-b6e0-0583623ad967-kube-api-access-t42nz\" (UniqueName: \"kubernetes.io/projected/ccdec0db-9dd5-4885-b6e0-0583623ad967-kube-api-access-t42nz\") pod \"ccdec0db-9dd5-4885-b6e0-0583623ad967\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") "
Oct 30 20:18:48.185184 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.184538    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-data\") pod \"ccdec0db-9dd5-4885-b6e0-0583623ad967\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") "
Oct 30 20:18:48.185184 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.184572    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-credential-socket\") pod \"ccdec0db-9dd5-4885-b6e0-0583623ad967\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") "
Oct 30 20:18:48.185184 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.184690    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-token\" (UniqueName: \"kubernetes.io/projected/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-token\") pod \"ccdec0db-9dd5-4885-b6e0-0583623ad967\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") "
Oct 30 20:18:48.185184 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.184831    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-envoy\") pod \"ccdec0db-9dd5-4885-b6e0-0583623ad967\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") "
Oct 30 20:18:48.185184 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.184884    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-workload-certs\") pod \"ccdec0db-9dd5-4885-b6e0-0583623ad967\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") "
Oct 30 20:18:48.185184 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.184938    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-podinfo\") pod \"ccdec0db-9dd5-4885-b6e0-0583623ad967\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") "
Oct 30 20:18:48.185184 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.184986    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccdec0db-9dd5-4885-b6e0-0583623ad967-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccdec0db-9dd5-4885-b6e0-0583623ad967-istiod-ca-cert\") pod \"ccdec0db-9dd5-4885-b6e0-0583623ad967\" (UID: \"ccdec0db-9dd5-4885-b6e0-0583623ad967\") "
Oct 30 20:18:48.186332 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.186070    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccdec0db-9dd5-4885-b6e0-0583623ad967-istiod-ca-cert" pod "ccdec0db-9dd5-4885-b6e0-0583623ad967" (UID: "ccdec0db-9dd5-4885-b6e0-0583623ad967"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:48.186641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.186525    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-workload-socket" pod "ccdec0db-9dd5-4885-b6e0-0583623ad967" (UID: "ccdec0db-9dd5-4885-b6e0-0583623ad967"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:48.187370 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.187342    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-data" pod "ccdec0db-9dd5-4885-b6e0-0583623ad967" (UID: "ccdec0db-9dd5-4885-b6e0-0583623ad967"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:48.187813 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.187786    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-credential-socket" pod "ccdec0db-9dd5-4885-b6e0-0583623ad967" (UID: "ccdec0db-9dd5-4885-b6e0-0583623ad967"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:48.187996 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.187974    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-workload-certs" pod "ccdec0db-9dd5-4885-b6e0-0583623ad967" (UID: "ccdec0db-9dd5-4885-b6e0-0583623ad967"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:48.189992 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.189719    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccdec0db-9dd5-4885-b6e0-0583623ad967-kube-api-access-t42nz" pod "ccdec0db-9dd5-4885-b6e0-0583623ad967" (UID: "ccdec0db-9dd5-4885-b6e0-0583623ad967"). InnerVolumeSpecName "kube-api-access-t42nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:48.190350 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.190329    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-token" pod "ccdec0db-9dd5-4885-b6e0-0583623ad967" (UID: "ccdec0db-9dd5-4885-b6e0-0583623ad967"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:48.192247 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.192218    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-envoy" pod "ccdec0db-9dd5-4885-b6e0-0583623ad967" (UID: "ccdec0db-9dd5-4885-b6e0-0583623ad967"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:48.192352 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.192246    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-podinfo" pod "ccdec0db-9dd5-4885-b6e0-0583623ad967" (UID: "ccdec0db-9dd5-4885-b6e0-0583623ad967"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:18:48.287202 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.287163    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:48.287202 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.287196    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:48.287427 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.287211    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:48.287427 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.287223    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccdec0db-9dd5-4885-b6e0-0583623ad967-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:48.287427 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.287234    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:48.287427 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.287246    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t42nz\" (UniqueName: \"kubernetes.io/projected/ccdec0db-9dd5-4885-b6e0-0583623ad967-kube-api-access-t42nz\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:48.287427 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.287258    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:48.287427 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.287272    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdec0db-9dd5-4885-b6e0-0583623ad967-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:48.287427 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:48.287283    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccdec0db-9dd5-4885-b6e0-0583623ad967-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:49.103897 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:49.103866    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"
Oct 30 20:18:49.161141 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:49.161110    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"]
Oct 30 20:18:49.165811 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:49.165788    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-hxf6t"]
Oct 30 20:18:49.200525 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:18:49.200496    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccdec0db-9dd5-4885-b6e0-0583623ad967" path="/var/lib/kubelet/pods/ccdec0db-9dd5-4885-b6e0-0583623ad967/volumes"
Oct 30 20:19:02.185157 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.185119    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"]
Oct 30 20:19:02.185976 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.185935    6030 cpu_manager.go:414] "containerMap: removing container" podUID="be081d10-b3c7-4a5a-9abf-9989ca04e364" containerName="istio-proxy"
Oct 30 20:19:02.185976 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.185962    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be081d10-b3c7-4a5a-9abf-9989ca04e364" containerName="istio-proxy"
Oct 30 20:19:02.186222 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.185991    6030 cpu_manager.go:414] "containerMap: removing container" podUID="ccdec0db-9dd5-4885-b6e0-0583623ad967" containerName="istio-proxy"
Oct 30 20:19:02.186222 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.185999    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccdec0db-9dd5-4885-b6e0-0583623ad967" containerName="istio-proxy"
Oct 30 20:19:02.186222 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.186009    6030 cpu_manager.go:414] "containerMap: removing container" podUID="ccdec0db-9dd5-4885-b6e0-0583623ad967" containerName="istio-proxy"
Oct 30 20:19:02.186222 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.186016    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccdec0db-9dd5-4885-b6e0-0583623ad967" containerName="istio-proxy"
Oct 30 20:19:02.186222 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.186202    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="be081d10-b3c7-4a5a-9abf-9989ca04e364" containerName="istio-proxy"
Oct 30 20:19:02.186222 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.186221    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccdec0db-9dd5-4885-b6e0-0583623ad967" containerName="istio-proxy"
Oct 30 20:19:02.186543 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.186230    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccdec0db-9dd5-4885-b6e0-0583623ad967" containerName="istio-proxy"
Oct 30 20:19:02.190627 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.190605    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.200226 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.198506    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-89c124\"/\"test-model-89c124-kserve-self-signed-certs\""
Oct 30 20:19:02.200226 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.198759    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-89c124\"/\"default-dockercfg-tmmdz\""
Oct 30 20:19:02.200226 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.199167    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"]
Oct 30 20:19:02.244452 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.244416    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-89c124\"/\"kube-root-ca.crt\""
Oct 30 20:19:02.244602 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.244483    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-89c124\"/\"openshift-service-ca.crt\""
Oct 30 20:19:02.275789 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.275751    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck2x5\" (UniqueName: \"kubernetes.io/projected/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-kube-api-access-ck2x5\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.275789 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.275789    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-dshm\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.276066 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.275983    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-model-cache\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.276173 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.276143    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-home\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.276220 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.276190    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-tmp-dir\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.276279 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.276225    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-tls-certs\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.383067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.377752    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-home\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.383067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.377807    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-tmp-dir\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.383067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.377840    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-tls-certs\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.383067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.377944    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck2x5\" (UniqueName: \"kubernetes.io/projected/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-kube-api-access-ck2x5\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.383067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.377982    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-dshm\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.383067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.378208    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-model-cache\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.383067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.378668    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-home\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.383067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.379158    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-tmp-dir\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.383067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.379951    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-model-cache\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.397739 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.397687    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-dshm\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.398190 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.398165    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-tls-certs\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.403149 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.401686    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck2x5\" (UniqueName: \"kubernetes.io/projected/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-kube-api-access-ck2x5\") pod \"test-model-89c124-kserve-54f686d44f-7mnb8\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") " pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.506856 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.506773    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:02.788426 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.788395    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"]
Oct 30 20:19:02.788706 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:19:02.788683    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5eb3ae64_0499_41d8_8f1b_f94f0ca43762.slice/crio-004277de25539d2a27ddcd6b487546a9014e1e538df811e3d7c6d993a15dfa8f WatchSource:0}: Error finding container 004277de25539d2a27ddcd6b487546a9014e1e538df811e3d7c6d993a15dfa8f: Status 404 returned error can't find the container with id 004277de25539d2a27ddcd6b487546a9014e1e538df811e3d7c6d993a15dfa8f
Oct 30 20:19:02.905918 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:02.905881    6030 scope.go:122] "RemoveContainer" containerID="7dd902950fb1647ef0caba07bdeff0b410ba9cd97a3a12e28ceb36328f767d5d"
Oct 30 20:19:03.210904 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:03.210869    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8" event={"ID":"5eb3ae64-0499-41d8-8f1b-f94f0ca43762","Type":"ContainerStarted","Data":"70c99136ca1e48a61f4a854f84429c9f781795c105afbf5e9e87ddbbda87c037"}
Oct 30 20:19:03.210904 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:03.210911    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8" event={"ID":"5eb3ae64-0499-41d8-8f1b-f94f0ca43762","Type":"ContainerStarted","Data":"004277de25539d2a27ddcd6b487546a9014e1e538df811e3d7c6d993a15dfa8f"}
Oct 30 20:19:03.234914 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:03.234865    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8" podStartSLOduration=1.23485112 podStartE2EDuration="1.23485112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:03.233247358 +0000 UTC m=+2822.148854215" watchObservedRunningTime="2026-10-30 20:19:03.23485112 +0000 UTC m=+2822.150457976"
Oct 30 20:19:03.507590 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:03.507505    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:04.288480 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:19:04.288439    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:19:12.507954 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:12.507920    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:12.523186 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:12.523155    6030 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:13.274478 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:19:13.274451    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:19:34.288643 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:19:34.288606    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:20:04.289135 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:20:04.289100    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:20:11.786647 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:11.786606    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"]
Oct 30 20:20:24.031175 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:24.031120    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"]
Oct 30 20:20:24.036827 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:24.036799    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd0074b-gshn6"]
Oct 30 20:20:25.201032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:25.200997    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a441f37d-5657-4821-bf5e-8a9c62e9eb86" path="/var/lib/kubelet/pods/a441f37d-5657-4821-bf5e-8a9c62e9eb86/volumes"
Oct 30 20:20:26.812218 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:26.812173    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj" podUID="f9310b6e-0c69-4e69-89a9-a268f132aa69" containerName="main" containerID="cri-o://fff00d669592485b50e4cf407b18033a58cd16fbfb9b99838919fbcd973590cd" gracePeriod=45
Oct 30 20:20:27.079990 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.079928    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:20:27.193159 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.193124    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-tmp-dir\") pod \"f9310b6e-0c69-4e69-89a9-a268f132aa69\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") "
Oct 30 20:20:27.193356 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.193223    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-model-cache\") pod \"f9310b6e-0c69-4e69-89a9-a268f132aa69\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") "
Oct 30 20:20:27.193356 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.193258    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-dshm\") pod \"f9310b6e-0c69-4e69-89a9-a268f132aa69\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") "
Oct 30 20:20:27.193356 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.193299    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-home\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-home\") pod \"f9310b6e-0c69-4e69-89a9-a268f132aa69\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") "
Oct 30 20:20:27.193527 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.193391    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9310b6e-0c69-4e69-89a9-a268f132aa69-kube-api-access-lkl62\" (UniqueName: \"kubernetes.io/projected/f9310b6e-0c69-4e69-89a9-a268f132aa69-kube-api-access-lkl62\") pod \"f9310b6e-0c69-4e69-89a9-a268f132aa69\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") "
Oct 30 20:20:27.193527 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.193468    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9310b6e-0c69-4e69-89a9-a268f132aa69-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9310b6e-0c69-4e69-89a9-a268f132aa69-tls-certs\") pod \"f9310b6e-0c69-4e69-89a9-a268f132aa69\" (UID: \"f9310b6e-0c69-4e69-89a9-a268f132aa69\") "
Oct 30 20:20:27.193647 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.193529    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-tmp-dir" pod "f9310b6e-0c69-4e69-89a9-a268f132aa69" (UID: "f9310b6e-0c69-4e69-89a9-a268f132aa69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:27.193848 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.193810    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-model-cache" pod "f9310b6e-0c69-4e69-89a9-a268f132aa69" (UID: "f9310b6e-0c69-4e69-89a9-a268f132aa69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:27.193993 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.193950    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:27.193993 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.193971    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:27.194487 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.194461    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-home" pod "f9310b6e-0c69-4e69-89a9-a268f132aa69" (UID: "f9310b6e-0c69-4e69-89a9-a268f132aa69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:27.195916 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.195895    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9310b6e-0c69-4e69-89a9-a268f132aa69-kube-api-access-lkl62" pod "f9310b6e-0c69-4e69-89a9-a268f132aa69" (UID: "f9310b6e-0c69-4e69-89a9-a268f132aa69"). InnerVolumeSpecName "kube-api-access-lkl62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:27.196339 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.196322    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9310b6e-0c69-4e69-89a9-a268f132aa69-tls-certs" pod "f9310b6e-0c69-4e69-89a9-a268f132aa69" (UID: "f9310b6e-0c69-4e69-89a9-a268f132aa69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:27.196418 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.196335    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-dshm" pod "f9310b6e-0c69-4e69-89a9-a268f132aa69" (UID: "f9310b6e-0c69-4e69-89a9-a268f132aa69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:27.295165 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.295117    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:27.295165 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.295150    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9310b6e-0c69-4e69-89a9-a268f132aa69-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:27.295165 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.295160    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkl62\" (UniqueName: \"kubernetes.io/projected/f9310b6e-0c69-4e69-89a9-a268f132aa69-kube-api-access-lkl62\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:27.295165 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.295171    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9310b6e-0c69-4e69-89a9-a268f132aa69-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:27.701881 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.701843    6030 generic.go:356] "Generic (PLEG): container finished" podID="f9310b6e-0c69-4e69-89a9-a268f132aa69" containerID="fff00d669592485b50e4cf407b18033a58cd16fbfb9b99838919fbcd973590cd" exitCode=0
Oct 30 20:20:27.702110 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.701898    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"
Oct 30 20:20:27.702110 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.701920    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj" event={"ID":"f9310b6e-0c69-4e69-89a9-a268f132aa69","Type":"ContainerDied","Data":"fff00d669592485b50e4cf407b18033a58cd16fbfb9b99838919fbcd973590cd"}
Oct 30 20:20:27.702110 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.701961    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj" event={"ID":"f9310b6e-0c69-4e69-89a9-a268f132aa69","Type":"ContainerDied","Data":"96c8a8a0704a4ad32ae360d96a41d7634ac29276d24e13dc136ad36db96e6951"}
Oct 30 20:20:27.702110 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.701982    6030 scope.go:122] "RemoveContainer" containerID="fff00d669592485b50e4cf407b18033a58cd16fbfb9b99838919fbcd973590cd"
Oct 30 20:20:27.711634 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.711608    6030 scope.go:122] "RemoveContainer" containerID="fff00d669592485b50e4cf407b18033a58cd16fbfb9b99838919fbcd973590cd"
Oct 30 20:20:27.711909 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:20:27.711891    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff00d669592485b50e4cf407b18033a58cd16fbfb9b99838919fbcd973590cd\": container with ID starting with fff00d669592485b50e4cf407b18033a58cd16fbfb9b99838919fbcd973590cd not found: ID does not exist" containerID="fff00d669592485b50e4cf407b18033a58cd16fbfb9b99838919fbcd973590cd"
Oct 30 20:20:27.712006 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.711917    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff00d669592485b50e4cf407b18033a58cd16fbfb9b99838919fbcd973590cd"} err="failed to get container status \"fff00d669592485b50e4cf407b18033a58cd16fbfb9b99838919fbcd973590cd\": rpc error: code = NotFound desc = could not find container \"fff00d669592485b50e4cf407b18033a58cd16fbfb9b99838919fbcd973590cd\": container with ID starting with fff00d669592485b50e4cf407b18033a58cd16fbfb9b99838919fbcd973590cd not found: ID does not exist"
Oct 30 20:20:27.723307 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.723284    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"]
Oct 30 20:20:27.727940 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:27.727918    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f63d20/test-model-f63d20-kserve-84bdf84565-276kj"]
Oct 30 20:20:29.201453 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:29.201422    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9310b6e-0c69-4e69-89a9-a268f132aa69" path="/var/lib/kubelet/pods/f9310b6e-0c69-4e69-89a9-a268f132aa69/volumes"
Oct 30 20:20:33.273132 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:33.273095    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"]
Oct 30 20:20:33.289112 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:33.289072    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"]
Oct 30 20:20:34.288636 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:20:34.288602    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:20:38.273928 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:38.273887    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn" podUID="914369a4-b049-4fa2-ab17-f31d0b21b083" containerName="payload-processing" containerID="cri-o://d85bc515b363adbaa0cbd2436c54dbf44db95ca437d10091417283c49d8ace3a" gracePeriod=25
Oct 30 20:20:38.290400 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:38.290363    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx" podUID="a083cbc5-d30b-4103-a5d8-a695e84418f2" containerName="payload-pre-processing" containerID="cri-o://201cbd61605c732ec6c398dbabbef339577c4dc904fd4fad394f858600939597" gracePeriod=25
Oct 30 20:20:45.381510 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:45.381468    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx" podUID="a083cbc5-d30b-4103-a5d8-a695e84418f2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:20:45.381510 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:45.381496    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn" podUID="914369a4-b049-4fa2-ab17-f31d0b21b083" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:20:53.279926 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:53.279880    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"]
Oct 30 20:20:55.381106 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:55.381068    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn" podUID="914369a4-b049-4fa2-ab17-f31d0b21b083" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:20:55.381576 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:55.381111    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx" podUID="a083cbc5-d30b-4103-a5d8-a695e84418f2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:20:58.560562 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.560527    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"
Oct 30 20:20:58.586465 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.586440    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"
Oct 30 20:20:58.666197 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.666168    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a083cbc5-d30b-4103-a5d8-a695e84418f2-kube-api-access-dlvpk\" (UniqueName: \"kubernetes.io/projected/a083cbc5-d30b-4103-a5d8-a695e84418f2-kube-api-access-dlvpk\") pod \"a083cbc5-d30b-4103-a5d8-a695e84418f2\" (UID: \"a083cbc5-d30b-4103-a5d8-a695e84418f2\") "
Oct 30 20:20:58.666389 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.666364    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a083cbc5-d30b-4103-a5d8-a695e84418f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a083cbc5-d30b-4103-a5d8-a695e84418f2-plugins-config-volume\") pod \"a083cbc5-d30b-4103-a5d8-a695e84418f2\" (UID: \"a083cbc5-d30b-4103-a5d8-a695e84418f2\") "
Oct 30 20:20:58.666460 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.666397    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914369a4-b049-4fa2-ab17-f31d0b21b083-kube-api-access-6md5q\" (UniqueName: \"kubernetes.io/projected/914369a4-b049-4fa2-ab17-f31d0b21b083-kube-api-access-6md5q\") pod \"914369a4-b049-4fa2-ab17-f31d0b21b083\" (UID: \"914369a4-b049-4fa2-ab17-f31d0b21b083\") "
Oct 30 20:20:58.666460 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.666428    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914369a4-b049-4fa2-ab17-f31d0b21b083-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914369a4-b049-4fa2-ab17-f31d0b21b083-plugins-config-volume\") pod \"914369a4-b049-4fa2-ab17-f31d0b21b083\" (UID: \"914369a4-b049-4fa2-ab17-f31d0b21b083\") "
Oct 30 20:20:58.666825 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.666793    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a083cbc5-d30b-4103-a5d8-a695e84418f2-plugins-config-volume" pod "a083cbc5-d30b-4103-a5d8-a695e84418f2" (UID: "a083cbc5-d30b-4103-a5d8-a695e84418f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:58.666963 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.666940    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a083cbc5-d30b-4103-a5d8-a695e84418f2-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:58.667068 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.666999    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914369a4-b049-4fa2-ab17-f31d0b21b083-plugins-config-volume" pod "914369a4-b049-4fa2-ab17-f31d0b21b083" (UID: "914369a4-b049-4fa2-ab17-f31d0b21b083"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:58.668790 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.668765    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914369a4-b049-4fa2-ab17-f31d0b21b083-kube-api-access-6md5q" pod "914369a4-b049-4fa2-ab17-f31d0b21b083" (UID: "914369a4-b049-4fa2-ab17-f31d0b21b083"). InnerVolumeSpecName "kube-api-access-6md5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:58.668909 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.668870    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a083cbc5-d30b-4103-a5d8-a695e84418f2-kube-api-access-dlvpk" pod "a083cbc5-d30b-4103-a5d8-a695e84418f2" (UID: "a083cbc5-d30b-4103-a5d8-a695e84418f2"). InnerVolumeSpecName "kube-api-access-dlvpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:58.768425 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.768392    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6md5q\" (UniqueName: \"kubernetes.io/projected/914369a4-b049-4fa2-ab17-f31d0b21b083-kube-api-access-6md5q\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:58.768425 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.768419    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914369a4-b049-4fa2-ab17-f31d0b21b083-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:58.768425 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.768429    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlvpk\" (UniqueName: \"kubernetes.io/projected/a083cbc5-d30b-4103-a5d8-a695e84418f2-kube-api-access-dlvpk\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:58.884706 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.884676    6030 generic.go:356] "Generic (PLEG): container finished" podID="914369a4-b049-4fa2-ab17-f31d0b21b083" containerID="d85bc515b363adbaa0cbd2436c54dbf44db95ca437d10091417283c49d8ace3a" exitCode=0
Oct 30 20:20:58.884894 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.884720    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"
Oct 30 20:20:58.884894 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.884756    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn" event={"ID":"914369a4-b049-4fa2-ab17-f31d0b21b083","Type":"ContainerDied","Data":"d85bc515b363adbaa0cbd2436c54dbf44db95ca437d10091417283c49d8ace3a"}
Oct 30 20:20:58.884894 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.884811    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn" event={"ID":"914369a4-b049-4fa2-ab17-f31d0b21b083","Type":"ContainerDied","Data":"fc45b9fc963d4069ade75c6a0912632832dbc470093f3f5468714721b7322676"}
Oct 30 20:20:58.884894 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.884833    6030 scope.go:122] "RemoveContainer" containerID="d85bc515b363adbaa0cbd2436c54dbf44db95ca437d10091417283c49d8ace3a"
Oct 30 20:20:58.886383 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.886359    6030 generic.go:356] "Generic (PLEG): container finished" podID="a083cbc5-d30b-4103-a5d8-a695e84418f2" containerID="201cbd61605c732ec6c398dbabbef339577c4dc904fd4fad394f858600939597" exitCode=0
Oct 30 20:20:58.886680 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.886398    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"
Oct 30 20:20:58.886680 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.886397    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx" event={"ID":"a083cbc5-d30b-4103-a5d8-a695e84418f2","Type":"ContainerDied","Data":"201cbd61605c732ec6c398dbabbef339577c4dc904fd4fad394f858600939597"}
Oct 30 20:20:58.886680 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.886524    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx" event={"ID":"a083cbc5-d30b-4103-a5d8-a695e84418f2","Type":"ContainerDied","Data":"d6eda001a19ca1b08ad422f3b0f72cfad7afe04ed5acfb63fefc4414660e7029"}
Oct 30 20:20:58.896485 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.896405    6030 scope.go:122] "RemoveContainer" containerID="d85bc515b363adbaa0cbd2436c54dbf44db95ca437d10091417283c49d8ace3a"
Oct 30 20:20:58.896734 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:20:58.896709    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d85bc515b363adbaa0cbd2436c54dbf44db95ca437d10091417283c49d8ace3a\": container with ID starting with d85bc515b363adbaa0cbd2436c54dbf44db95ca437d10091417283c49d8ace3a not found: ID does not exist" containerID="d85bc515b363adbaa0cbd2436c54dbf44db95ca437d10091417283c49d8ace3a"
Oct 30 20:20:58.896840 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.896739    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d85bc515b363adbaa0cbd2436c54dbf44db95ca437d10091417283c49d8ace3a"} err="failed to get container status \"d85bc515b363adbaa0cbd2436c54dbf44db95ca437d10091417283c49d8ace3a\": rpc error: code = NotFound desc = could not find container \"d85bc515b363adbaa0cbd2436c54dbf44db95ca437d10091417283c49d8ace3a\": container with ID starting with d85bc515b363adbaa0cbd2436c54dbf44db95ca437d10091417283c49d8ace3a not found: ID does not exist"
Oct 30 20:20:58.896840 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.896757    6030 scope.go:122] "RemoveContainer" containerID="201cbd61605c732ec6c398dbabbef339577c4dc904fd4fad394f858600939597"
Oct 30 20:20:58.905957 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.905942    6030 scope.go:122] "RemoveContainer" containerID="201cbd61605c732ec6c398dbabbef339577c4dc904fd4fad394f858600939597"
Oct 30 20:20:58.906249 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:20:58.906227    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201cbd61605c732ec6c398dbabbef339577c4dc904fd4fad394f858600939597\": container with ID starting with 201cbd61605c732ec6c398dbabbef339577c4dc904fd4fad394f858600939597 not found: ID does not exist" containerID="201cbd61605c732ec6c398dbabbef339577c4dc904fd4fad394f858600939597"
Oct 30 20:20:58.906350 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.906253    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201cbd61605c732ec6c398dbabbef339577c4dc904fd4fad394f858600939597"} err="failed to get container status \"201cbd61605c732ec6c398dbabbef339577c4dc904fd4fad394f858600939597\": rpc error: code = NotFound desc = could not find container \"201cbd61605c732ec6c398dbabbef339577c4dc904fd4fad394f858600939597\": container with ID starting with 201cbd61605c732ec6c398dbabbef339577c4dc904fd4fad394f858600939597 not found: ID does not exist"
Oct 30 20:20:58.911407 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.911386    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"]
Oct 30 20:20:58.916732 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.916707    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f63d20-79ddcbd5c8-dsbmn"]
Oct 30 20:20:58.931030 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.930995    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"]
Oct 30 20:20:58.932978 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:58.932956    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f63d20-64dfb99747-wrctx"]
Oct 30 20:20:59.200274 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:59.200197    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914369a4-b049-4fa2-ab17-f31d0b21b083" path="/var/lib/kubelet/pods/914369a4-b049-4fa2-ab17-f31d0b21b083/volumes"
Oct 30 20:20:59.200632 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:20:59.200619    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a083cbc5-d30b-4103-a5d8-a695e84418f2" path="/var/lib/kubelet/pods/a083cbc5-d30b-4103-a5d8-a695e84418f2/volumes"
Oct 30 20:21:04.289564 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:21:04.289527    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:21:05.921665 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:05.921621    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"]
Oct 30 20:21:08.312550 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.312491    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8" podUID="5eb3ae64-0499-41d8-8f1b-f94f0ca43762" containerName="main" containerID="cri-o://70c99136ca1e48a61f4a854f84429c9f781795c105afbf5e9e87ddbbda87c037" gracePeriod=45
Oct 30 20:21:08.582630 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.582600    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:21:08.683493 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.683422    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-model-cache\") pod \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") "
Oct 30 20:21:08.683666 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.683507    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-tmp-dir\") pod \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") "
Oct 30 20:21:08.683666 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.683525    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-tls-certs\" (UniqueName: \"kubernetes.io/secret/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-tls-certs\") pod \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") "
Oct 30 20:21:08.683666 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.683603    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-home\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-home\") pod \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") "
Oct 30 20:21:08.683857 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.683677    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-kube-api-access-ck2x5\" (UniqueName: \"kubernetes.io/projected/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-kube-api-access-ck2x5\") pod \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") "
Oct 30 20:21:08.683857 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.683709    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-dshm\") pod \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\" (UID: \"5eb3ae64-0499-41d8-8f1b-f94f0ca43762\") "
Oct 30 20:21:08.684021 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.683702    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-model-cache" pod "5eb3ae64-0499-41d8-8f1b-f94f0ca43762" (UID: "5eb3ae64-0499-41d8-8f1b-f94f0ca43762"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:08.684125 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.683796    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-tmp-dir" pod "5eb3ae64-0499-41d8-8f1b-f94f0ca43762" (UID: "5eb3ae64-0499-41d8-8f1b-f94f0ca43762"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:08.684125 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.683872    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-home" pod "5eb3ae64-0499-41d8-8f1b-f94f0ca43762" (UID: "5eb3ae64-0499-41d8-8f1b-f94f0ca43762"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:08.684406 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.684347    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:08.684406 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.684372    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:08.684406 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.684385    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:08.686572 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.686549    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-tls-certs" pod "5eb3ae64-0499-41d8-8f1b-f94f0ca43762" (UID: "5eb3ae64-0499-41d8-8f1b-f94f0ca43762"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:08.686572 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.686553    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-kube-api-access-ck2x5" pod "5eb3ae64-0499-41d8-8f1b-f94f0ca43762" (UID: "5eb3ae64-0499-41d8-8f1b-f94f0ca43762"). InnerVolumeSpecName "kube-api-access-ck2x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:08.686706 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.686594    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-dshm" pod "5eb3ae64-0499-41d8-8f1b-f94f0ca43762" (UID: "5eb3ae64-0499-41d8-8f1b-f94f0ca43762"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:08.784995 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.784968    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:08.784995 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.784996    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck2x5\" (UniqueName: \"kubernetes.io/projected/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-kube-api-access-ck2x5\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:08.785197 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.785007    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eb3ae64-0499-41d8-8f1b-f94f0ca43762-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:08.945301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.945268    6030 generic.go:356] "Generic (PLEG): container finished" podID="5eb3ae64-0499-41d8-8f1b-f94f0ca43762" containerID="70c99136ca1e48a61f4a854f84429c9f781795c105afbf5e9e87ddbbda87c037" exitCode=0
Oct 30 20:21:08.945523 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.945314    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"
Oct 30 20:21:08.945523 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.945348    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8" event={"ID":"5eb3ae64-0499-41d8-8f1b-f94f0ca43762","Type":"ContainerDied","Data":"70c99136ca1e48a61f4a854f84429c9f781795c105afbf5e9e87ddbbda87c037"}
Oct 30 20:21:08.945523 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.945395    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8" event={"ID":"5eb3ae64-0499-41d8-8f1b-f94f0ca43762","Type":"ContainerDied","Data":"004277de25539d2a27ddcd6b487546a9014e1e538df811e3d7c6d993a15dfa8f"}
Oct 30 20:21:08.945523 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.945417    6030 scope.go:122] "RemoveContainer" containerID="70c99136ca1e48a61f4a854f84429c9f781795c105afbf5e9e87ddbbda87c037"
Oct 30 20:21:08.955110 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.955090    6030 scope.go:122] "RemoveContainer" containerID="70c99136ca1e48a61f4a854f84429c9f781795c105afbf5e9e87ddbbda87c037"
Oct 30 20:21:08.955360 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:21:08.955342    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c99136ca1e48a61f4a854f84429c9f781795c105afbf5e9e87ddbbda87c037\": container with ID starting with 70c99136ca1e48a61f4a854f84429c9f781795c105afbf5e9e87ddbbda87c037 not found: ID does not exist" containerID="70c99136ca1e48a61f4a854f84429c9f781795c105afbf5e9e87ddbbda87c037"
Oct 30 20:21:08.955427 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.955368    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c99136ca1e48a61f4a854f84429c9f781795c105afbf5e9e87ddbbda87c037"} err="failed to get container status \"70c99136ca1e48a61f4a854f84429c9f781795c105afbf5e9e87ddbbda87c037\": rpc error: code = NotFound desc = could not find container \"70c99136ca1e48a61f4a854f84429c9f781795c105afbf5e9e87ddbbda87c037\": container with ID starting with 70c99136ca1e48a61f4a854f84429c9f781795c105afbf5e9e87ddbbda87c037 not found: ID does not exist"
Oct 30 20:21:08.969597 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.969566    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"]
Oct 30 20:21:08.976761 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:08.976737    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-89c124/test-model-89c124-kserve-54f686d44f-7mnb8"]
Oct 30 20:21:09.202378 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:09.202285    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb3ae64-0499-41d8-8f1b-f94f0ca43762" path="/var/lib/kubelet/pods/5eb3ae64-0499-41d8-8f1b-f94f0ca43762/volumes"
Oct 30 20:21:10.922879 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:10.922835    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt" podUID="fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0" containerName="payload-processing" containerID="cri-o://69e29ab39d57e55a2244aff106d8ec18521f9ea00d7bd6e1cd45362aa9931cb8" gracePeriod=25
Oct 30 20:21:11.180676 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:11.180621    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"
Oct 30 20:21:11.214332 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:11.214307    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0-plugins-config-volume\") pod \"fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0\" (UID: \"fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0\") "
Oct 30 20:21:11.214502 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:11.214423    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0-kube-api-access-vstlw\" (UniqueName: \"kubernetes.io/projected/fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0-kube-api-access-vstlw\") pod \"fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0\" (UID: \"fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0\") "
Oct 30 20:21:11.214842 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:11.214813    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0-plugins-config-volume" pod "fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0" (UID: "fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:11.218279 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:11.218249    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0-kube-api-access-vstlw" pod "fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0" (UID: "fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0"). InnerVolumeSpecName "kube-api-access-vstlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:11.316093 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:11.316038    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:11.316093 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:11.316089    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vstlw\" (UniqueName: \"kubernetes.io/projected/fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0-kube-api-access-vstlw\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:11.965796 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:11.965755    6030 generic.go:356] "Generic (PLEG): container finished" podID="fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0" containerID="69e29ab39d57e55a2244aff106d8ec18521f9ea00d7bd6e1cd45362aa9931cb8" exitCode=0
Oct 30 20:21:11.966425 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:11.965848    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"
Oct 30 20:21:11.966425 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:11.965911    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt" event={"ID":"fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0","Type":"ContainerDied","Data":"69e29ab39d57e55a2244aff106d8ec18521f9ea00d7bd6e1cd45362aa9931cb8"}
Oct 30 20:21:11.966425 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:11.966019    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt" event={"ID":"fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0","Type":"ContainerDied","Data":"86ffd51c35b7482c9a3fe93999f13322196ff1fb0f8816a1b0332d84f2b3093f"}
Oct 30 20:21:11.966425 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:11.966078    6030 scope.go:122] "RemoveContainer" containerID="69e29ab39d57e55a2244aff106d8ec18521f9ea00d7bd6e1cd45362aa9931cb8"
Oct 30 20:21:11.981719 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:11.981695    6030 scope.go:122] "RemoveContainer" containerID="69e29ab39d57e55a2244aff106d8ec18521f9ea00d7bd6e1cd45362aa9931cb8"
Oct 30 20:21:11.982170 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:21:11.982106    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69e29ab39d57e55a2244aff106d8ec18521f9ea00d7bd6e1cd45362aa9931cb8\": container with ID starting with 69e29ab39d57e55a2244aff106d8ec18521f9ea00d7bd6e1cd45362aa9931cb8 not found: ID does not exist" containerID="69e29ab39d57e55a2244aff106d8ec18521f9ea00d7bd6e1cd45362aa9931cb8"
Oct 30 20:21:11.982170 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:11.982148    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69e29ab39d57e55a2244aff106d8ec18521f9ea00d7bd6e1cd45362aa9931cb8"} err="failed to get container status \"69e29ab39d57e55a2244aff106d8ec18521f9ea00d7bd6e1cd45362aa9931cb8\": rpc error: code = NotFound desc = could not find container \"69e29ab39d57e55a2244aff106d8ec18521f9ea00d7bd6e1cd45362aa9931cb8\": container with ID starting with 69e29ab39d57e55a2244aff106d8ec18521f9ea00d7bd6e1cd45362aa9931cb8 not found: ID does not exist"
Oct 30 20:21:12.002012 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:12.001984    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"]
Oct 30 20:21:12.010155 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:12.010129    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c257e7-d4df69d7b-5s4bt"]
Oct 30 20:21:13.200655 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:13.200626    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0" path="/var/lib/kubelet/pods/fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0/volumes"
Oct 30 20:21:24.950964 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:24.950927    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"]
Oct 30 20:21:24.951530 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:24.951188    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2" podUID="6484295e-f0bf-4194-9416-12c15129dd4a" containerName="istio-proxy" containerID="cri-o://c87e8156656784a350cdde7d73ae6b4d785111a89281b1152d43aa55f67ad99c" gracePeriod=30
Oct 30 20:21:25.365363 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:25.365270    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"]
Oct 30 20:21:25.365545 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:25.365515    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz" podUID="0f6f8eab-4484-4542-9e49-9b13bb4a8cef" containerName="maas-api" containerID="cri-o://779529070ea31bb9a6bec4dc8e8225766c36f6e1e7025b06128f9e8bc31b1058" gracePeriod=30
Oct 30 20:21:25.838581 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:25.838551    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:21:25.982724 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:25.982638    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-openshift-service-ca\") pod \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\" (UID: \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\") "
Oct 30 20:21:25.983220 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:25.982789    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-kube-api-access-54z2z\" (UniqueName: \"kubernetes.io/projected/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-kube-api-access-54z2z\") pod \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\" (UID: \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\") "
Oct 30 20:21:25.983220 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:25.982849    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-metrics-certs\") pod \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\" (UID: \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\") "
Oct 30 20:21:25.983220 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:25.982874    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-maas-api-tls\") pod \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\" (UID: \"0f6f8eab-4484-4542-9e49-9b13bb4a8cef\") "
Oct 30 20:21:25.983220 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:25.983037    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-openshift-service-ca" pod "0f6f8eab-4484-4542-9e49-9b13bb4a8cef" (UID: "0f6f8eab-4484-4542-9e49-9b13bb4a8cef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:25.983465 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:25.983252    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:25.985235 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:25.985208    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-maas-api-tls" pod "0f6f8eab-4484-4542-9e49-9b13bb4a8cef" (UID: "0f6f8eab-4484-4542-9e49-9b13bb4a8cef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:25.985705 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:25.985676    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-metrics-certs" pod "0f6f8eab-4484-4542-9e49-9b13bb4a8cef" (UID: "0f6f8eab-4484-4542-9e49-9b13bb4a8cef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:25.985806 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:25.985716    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-kube-api-access-54z2z" pod "0f6f8eab-4484-4542-9e49-9b13bb4a8cef" (UID: "0f6f8eab-4484-4542-9e49-9b13bb4a8cef"). InnerVolumeSpecName "kube-api-access-54z2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:26.056451 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:26.056416    6030 generic.go:356] "Generic (PLEG): container finished" podID="0f6f8eab-4484-4542-9e49-9b13bb4a8cef" containerID="779529070ea31bb9a6bec4dc8e8225766c36f6e1e7025b06128f9e8bc31b1058" exitCode=0
Oct 30 20:21:26.056654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:26.056467    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"
Oct 30 20:21:26.056654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:26.056499    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz" event={"ID":"0f6f8eab-4484-4542-9e49-9b13bb4a8cef","Type":"ContainerDied","Data":"779529070ea31bb9a6bec4dc8e8225766c36f6e1e7025b06128f9e8bc31b1058"}
Oct 30 20:21:26.056654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:26.056568    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz" event={"ID":"0f6f8eab-4484-4542-9e49-9b13bb4a8cef","Type":"ContainerDied","Data":"9c82851b7e7b92a3660e23f123aa6899472b2affe4c47b16263e0f358cc381a2"}
Oct 30 20:21:26.056654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:26.056598    6030 scope.go:122] "RemoveContainer" containerID="779529070ea31bb9a6bec4dc8e8225766c36f6e1e7025b06128f9e8bc31b1058"
Oct 30 20:21:26.066480 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:26.066459    6030 scope.go:122] "RemoveContainer" containerID="779529070ea31bb9a6bec4dc8e8225766c36f6e1e7025b06128f9e8bc31b1058"
Oct 30 20:21:26.066831 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:21:26.066810    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779529070ea31bb9a6bec4dc8e8225766c36f6e1e7025b06128f9e8bc31b1058\": container with ID starting with 779529070ea31bb9a6bec4dc8e8225766c36f6e1e7025b06128f9e8bc31b1058 not found: ID does not exist" containerID="779529070ea31bb9a6bec4dc8e8225766c36f6e1e7025b06128f9e8bc31b1058"
Oct 30 20:21:26.066911 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:26.066836    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779529070ea31bb9a6bec4dc8e8225766c36f6e1e7025b06128f9e8bc31b1058"} err="failed to get container status \"779529070ea31bb9a6bec4dc8e8225766c36f6e1e7025b06128f9e8bc31b1058\": rpc error: code = NotFound desc = could not find container \"779529070ea31bb9a6bec4dc8e8225766c36f6e1e7025b06128f9e8bc31b1058\": container with ID starting with 779529070ea31bb9a6bec4dc8e8225766c36f6e1e7025b06128f9e8bc31b1058 not found: ID does not exist"
Oct 30 20:21:26.084387 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:26.084364    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54z2z\" (UniqueName: \"kubernetes.io/projected/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-kube-api-access-54z2z\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:26.084491 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:26.084390    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:26.084491 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:26.084400    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6f8eab-4484-4542-9e49-9b13bb4a8cef-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:26.093290 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:26.093267    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"]
Oct 30 20:21:26.101645 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:26.101619    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-89c124-6794f67f4b-h8sbz"]
Oct 30 20:21:27.200163 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:27.200131    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f6f8eab-4484-4542-9e49-9b13bb4a8cef" path="/var/lib/kubelet/pods/0f6f8eab-4484-4542-9e49-9b13bb4a8cef/volumes"
Oct 30 20:21:30.084551 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.084517    6030 generic.go:356] "Generic (PLEG): container finished" podID="6484295e-f0bf-4194-9416-12c15129dd4a" containerID="c87e8156656784a350cdde7d73ae6b4d785111a89281b1152d43aa55f67ad99c" exitCode=0
Oct 30 20:21:30.085013 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.084589    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2" event={"ID":"6484295e-f0bf-4194-9416-12c15129dd4a","Type":"ContainerDied","Data":"c87e8156656784a350cdde7d73ae6b4d785111a89281b1152d43aa55f67ad99c"}
Oct 30 20:21:30.085013 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.084661    6030 scope.go:122] "RemoveContainer" containerID="9c46eb3acfb2f2da869e5cd6f8096318a029748e3ee24ea055bcdeb0cb05cccf"
Oct 30 20:21:30.215059 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.215023    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:21:30.330688 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.330595    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6484295e-f0bf-4194-9416-12c15129dd4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6484295e-f0bf-4194-9416-12c15129dd4a-istiod-ca-cert\") pod \"6484295e-f0bf-4194-9416-12c15129dd4a\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") "
Oct 30 20:21:30.330688 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.330643    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-workload-certs\") pod \"6484295e-f0bf-4194-9416-12c15129dd4a\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") "
Oct 30 20:21:30.330688 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.330673    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-istio-data\") pod \"6484295e-f0bf-4194-9416-12c15129dd4a\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") "
Oct 30 20:21:30.330993 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.330721    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-istio-envoy\") pod \"6484295e-f0bf-4194-9416-12c15129dd4a\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") "
Oct 30 20:21:30.330993 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.330748    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6484295e-f0bf-4194-9416-12c15129dd4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6484295e-f0bf-4194-9416-12c15129dd4a-istio-podinfo\") pod \"6484295e-f0bf-4194-9416-12c15129dd4a\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") "
Oct 30 20:21:30.330993 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.330826    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-credential-socket\") pod \"6484295e-f0bf-4194-9416-12c15129dd4a\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") "
Oct 30 20:21:30.330993 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.330882    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-workload-socket\") pod \"6484295e-f0bf-4194-9416-12c15129dd4a\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") "
Oct 30 20:21:30.331244 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.331006    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6484295e-f0bf-4194-9416-12c15129dd4a-istio-token\" (UniqueName: \"kubernetes.io/projected/6484295e-f0bf-4194-9416-12c15129dd4a-istio-token\") pod \"6484295e-f0bf-4194-9416-12c15129dd4a\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") "
Oct 30 20:21:30.331244 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.331089    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6484295e-f0bf-4194-9416-12c15129dd4a-kube-api-access-qfn96\" (UniqueName: \"kubernetes.io/projected/6484295e-f0bf-4194-9416-12c15129dd4a-kube-api-access-qfn96\") pod \"6484295e-f0bf-4194-9416-12c15129dd4a\" (UID: \"6484295e-f0bf-4194-9416-12c15129dd4a\") "
Oct 30 20:21:30.331244 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.331097    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-istio-data" pod "6484295e-f0bf-4194-9416-12c15129dd4a" (UID: "6484295e-f0bf-4194-9416-12c15129dd4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:30.331244 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.331133    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6484295e-f0bf-4194-9416-12c15129dd4a-istiod-ca-cert" pod "6484295e-f0bf-4194-9416-12c15129dd4a" (UID: "6484295e-f0bf-4194-9416-12c15129dd4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:30.331450 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.331228    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-workload-certs" pod "6484295e-f0bf-4194-9416-12c15129dd4a" (UID: "6484295e-f0bf-4194-9416-12c15129dd4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:30.331450 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.331274    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-credential-socket" pod "6484295e-f0bf-4194-9416-12c15129dd4a" (UID: "6484295e-f0bf-4194-9416-12c15129dd4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:30.331450 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.331346    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-workload-socket" pod "6484295e-f0bf-4194-9416-12c15129dd4a" (UID: "6484295e-f0bf-4194-9416-12c15129dd4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:30.331606 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.331586    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6484295e-f0bf-4194-9416-12c15129dd4a-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:30.331677 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.331609    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:30.331677 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.331624    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:30.331677 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.331640    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:30.331677 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.331654    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:30.333478 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.333451    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-istio-envoy" pod "6484295e-f0bf-4194-9416-12c15129dd4a" (UID: "6484295e-f0bf-4194-9416-12c15129dd4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:30.333596 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.333471    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6484295e-f0bf-4194-9416-12c15129dd4a-istio-token" pod "6484295e-f0bf-4194-9416-12c15129dd4a" (UID: "6484295e-f0bf-4194-9416-12c15129dd4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:30.333596 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.333488    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6484295e-f0bf-4194-9416-12c15129dd4a-istio-podinfo" pod "6484295e-f0bf-4194-9416-12c15129dd4a" (UID: "6484295e-f0bf-4194-9416-12c15129dd4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:21:30.333596 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.333540    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6484295e-f0bf-4194-9416-12c15129dd4a-kube-api-access-qfn96" pod "6484295e-f0bf-4194-9416-12c15129dd4a" (UID: "6484295e-f0bf-4194-9416-12c15129dd4a"). InnerVolumeSpecName "kube-api-access-qfn96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:30.433169 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.433136    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6484295e-f0bf-4194-9416-12c15129dd4a-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:30.433169 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.433163    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfn96\" (UniqueName: \"kubernetes.io/projected/6484295e-f0bf-4194-9416-12c15129dd4a-kube-api-access-qfn96\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:30.433169 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.433173    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6484295e-f0bf-4194-9416-12c15129dd4a-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:30.433420 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:30.433183    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6484295e-f0bf-4194-9416-12c15129dd4a-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:31.092059 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:31.092025    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"
Oct 30 20:21:31.092530 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:31.092084    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2" event={"ID":"6484295e-f0bf-4194-9416-12c15129dd4a","Type":"ContainerDied","Data":"9c2a6af9116bacdbbe46f8d2fb83092d9bd61fdbcc79aa51ac1dc375547974ad"}
Oct 30 20:21:31.092530 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:31.092143    6030 scope.go:122] "RemoveContainer" containerID="c87e8156656784a350cdde7d73ae6b4d785111a89281b1152d43aa55f67ad99c"
Oct 30 20:21:31.118425 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:31.118400    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"]
Oct 30 20:21:31.123341 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:31.123319    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c257e7-openshift-default-f68cbf4f5-4vvk2"]
Oct 30 20:21:31.201262 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:31.201232    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6484295e-f0bf-4194-9416-12c15129dd4a" path="/var/lib/kubelet/pods/6484295e-f0bf-4194-9416-12c15129dd4a/volumes"
Oct 30 20:21:32.897949 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:32.897912    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"]
Oct 30 20:21:32.898494 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:32.898181    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb" podUID="c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e" containerName="discovery" containerID="cri-o://bad7893287a4c88918507b1731cdea2bc8400d312a1879d3a369125426c87527" gracePeriod=30
Oct 30 20:21:33.106891 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.106861    6030 generic.go:356] "Generic (PLEG): container finished" podID="c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e" containerID="bad7893287a4c88918507b1731cdea2bc8400d312a1879d3a369125426c87527" exitCode=0
Oct 30 20:21:33.107080 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.106944    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb" event={"ID":"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e","Type":"ContainerDied","Data":"bad7893287a4c88918507b1731cdea2bc8400d312a1879d3a369125426c87527"}
Oct 30 20:21:33.150680 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.150626    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:21:33.266594 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.266559    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-cacerts\" (UniqueName: \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-cacerts\") pod \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") "
Oct 30 20:21:33.266787 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.266660    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-token\" (UniqueName: \"kubernetes.io/projected/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-token\") pod \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") "
Oct 30 20:21:33.266787 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.266720    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-kubeconfig\") pod \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") "
Oct 30 20:21:33.266924 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.266803    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-csr-dns-cert\") pod \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") "
Oct 30 20:21:33.266924 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.266844    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-local-certs\") pod \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") "
Oct 30 20:21:33.267035 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.266926    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-kube-api-access-n8p2q\" (UniqueName: \"kubernetes.io/projected/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-kube-api-access-n8p2q\") pod \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") "
Oct 30 20:21:33.267035 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.266957    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-csr-ca-configmap\") pod \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\" (UID: \"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e\") "
Oct 30 20:21:33.267638 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.267604    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-csr-ca-configmap" pod "c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e" (UID: "c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:33.269219 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.269189    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-kubeconfig" pod "c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e" (UID: "c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:33.269527 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.269498    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-token" pod "c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e" (UID: "c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:33.269617 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.269556    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-cacerts" pod "c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e" (UID: "c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:33.269617 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.269575    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-csr-dns-cert" pod "c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e" (UID: "c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:33.269617 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.269587    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-local-certs" pod "c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e" (UID: "c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:33.269796 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.269779    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-kube-api-access-n8p2q" pod "c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e" (UID: "c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e"). InnerVolumeSpecName "kube-api-access-n8p2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:33.368308 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.368272    6030 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-csr-dns-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:33.368308 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.368305    6030 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-local-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:33.368525 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.368318    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8p2q\" (UniqueName: \"kubernetes.io/projected/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-kube-api-access-n8p2q\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:33.368525 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.368332    6030 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-csr-ca-configmap\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:33.368525 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.368345    6030 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-cacerts\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:33.368525 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.368357    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:33.368525 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:33.368368    6030 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e-istio-kubeconfig\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:34.113981 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:34.113952    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"
Oct 30 20:21:34.114470 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:34.113978    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb" event={"ID":"c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e","Type":"ContainerDied","Data":"748243a8bde0bcb70cd9e385a57a1706256f16f009f9933185401e06b7888ab1"}
Oct 30 20:21:34.114470 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:34.114073    6030 scope.go:122] "RemoveContainer" containerID="bad7893287a4c88918507b1731cdea2bc8400d312a1879d3a369125426c87527"
Oct 30 20:21:34.147115 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:34.147090    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"]
Oct 30 20:21:34.153809 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:34.153789    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9msfb"]
Oct 30 20:21:34.287771 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:21:34.287738    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:21:35.200265 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:35.200232    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e" path="/var/lib/kubelet/pods/c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e/volumes"
Oct 30 20:21:42.623745 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:42.623667    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"]
Oct 30 20:21:42.624195 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:42.624079    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l" podUID="86138fcf-3364-46db-ab0b-9c327868687a" containerName="maas-api" containerID="cri-o://0aa922e28d25a40894dcbeb388488fbb4353eb609099f682f6fac4918361c838" gracePeriod=30
Oct 30 20:21:42.935067 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:42.930772    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:21:42.968508 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:42.968476    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86138fcf-3364-46db-ab0b-9c327868687a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86138fcf-3364-46db-ab0b-9c327868687a-maas-api-tls\") pod \"86138fcf-3364-46db-ab0b-9c327868687a\" (UID: \"86138fcf-3364-46db-ab0b-9c327868687a\") "
Oct 30 20:21:42.968695 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:42.968622    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86138fcf-3364-46db-ab0b-9c327868687a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86138fcf-3364-46db-ab0b-9c327868687a-openshift-service-ca\") pod \"86138fcf-3364-46db-ab0b-9c327868687a\" (UID: \"86138fcf-3364-46db-ab0b-9c327868687a\") "
Oct 30 20:21:42.968695 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:42.968676    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86138fcf-3364-46db-ab0b-9c327868687a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/86138fcf-3364-46db-ab0b-9c327868687a-metrics-certs\") pod \"86138fcf-3364-46db-ab0b-9c327868687a\" (UID: \"86138fcf-3364-46db-ab0b-9c327868687a\") "
Oct 30 20:21:42.968838 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:42.968710    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86138fcf-3364-46db-ab0b-9c327868687a-kube-api-access-hs655\" (UniqueName: \"kubernetes.io/projected/86138fcf-3364-46db-ab0b-9c327868687a-kube-api-access-hs655\") pod \"86138fcf-3364-46db-ab0b-9c327868687a\" (UID: \"86138fcf-3364-46db-ab0b-9c327868687a\") "
Oct 30 20:21:42.969018 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:42.968995    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86138fcf-3364-46db-ab0b-9c327868687a-openshift-service-ca" pod "86138fcf-3364-46db-ab0b-9c327868687a" (UID: "86138fcf-3364-46db-ab0b-9c327868687a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:42.969208 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:42.969192    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86138fcf-3364-46db-ab0b-9c327868687a-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:42.971686 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:42.971652    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86138fcf-3364-46db-ab0b-9c327868687a-maas-api-tls" pod "86138fcf-3364-46db-ab0b-9c327868687a" (UID: "86138fcf-3364-46db-ab0b-9c327868687a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:42.971808 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:42.971722    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86138fcf-3364-46db-ab0b-9c327868687a-metrics-certs" pod "86138fcf-3364-46db-ab0b-9c327868687a" (UID: "86138fcf-3364-46db-ab0b-9c327868687a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:42.971808 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:42.971761    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86138fcf-3364-46db-ab0b-9c327868687a-kube-api-access-hs655" pod "86138fcf-3364-46db-ab0b-9c327868687a" (UID: "86138fcf-3364-46db-ab0b-9c327868687a"). InnerVolumeSpecName "kube-api-access-hs655". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:43.070807 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:43.070772    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86138fcf-3364-46db-ab0b-9c327868687a-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:43.070807 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:43.070801    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs655\" (UniqueName: \"kubernetes.io/projected/86138fcf-3364-46db-ab0b-9c327868687a-kube-api-access-hs655\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:43.070807 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:43.070812    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86138fcf-3364-46db-ab0b-9c327868687a-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:43.171581 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:43.170646    6030 generic.go:356] "Generic (PLEG): container finished" podID="86138fcf-3364-46db-ab0b-9c327868687a" containerID="0aa922e28d25a40894dcbeb388488fbb4353eb609099f682f6fac4918361c838" exitCode=0
Oct 30 20:21:43.171581 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:43.170689    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l" event={"ID":"86138fcf-3364-46db-ab0b-9c327868687a","Type":"ContainerDied","Data":"0aa922e28d25a40894dcbeb388488fbb4353eb609099f682f6fac4918361c838"}
Oct 30 20:21:43.171581 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:43.170728    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"
Oct 30 20:21:43.171581 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:43.170806    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l" event={"ID":"86138fcf-3364-46db-ab0b-9c327868687a","Type":"ContainerDied","Data":"188e9e60f815b1e2bebaa2e5e4b7f106e475ab865d40c0b729479597d0516bc9"}
Oct 30 20:21:43.171581 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:43.170841    6030 scope.go:122] "RemoveContainer" containerID="0aa922e28d25a40894dcbeb388488fbb4353eb609099f682f6fac4918361c838"
Oct 30 20:21:43.181769 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:43.181752    6030 scope.go:122] "RemoveContainer" containerID="0aa922e28d25a40894dcbeb388488fbb4353eb609099f682f6fac4918361c838"
Oct 30 20:21:43.182073 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:21:43.182024    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa922e28d25a40894dcbeb388488fbb4353eb609099f682f6fac4918361c838\": container with ID starting with 0aa922e28d25a40894dcbeb388488fbb4353eb609099f682f6fac4918361c838 not found: ID does not exist" containerID="0aa922e28d25a40894dcbeb388488fbb4353eb609099f682f6fac4918361c838"
Oct 30 20:21:43.182128 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:43.182093    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa922e28d25a40894dcbeb388488fbb4353eb609099f682f6fac4918361c838"} err="failed to get container status \"0aa922e28d25a40894dcbeb388488fbb4353eb609099f682f6fac4918361c838\": rpc error: code = NotFound desc = could not find container \"0aa922e28d25a40894dcbeb388488fbb4353eb609099f682f6fac4918361c838\": container with ID starting with 0aa922e28d25a40894dcbeb388488fbb4353eb609099f682f6fac4918361c838 not found: ID does not exist"
Oct 30 20:21:43.201229 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:43.201209    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"]
Oct 30 20:21:43.201350 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:43.201240    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-517027-7c57c8fb47-96x9l"]
Oct 30 20:21:45.200206 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:45.200169    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86138fcf-3364-46db-ab0b-9c327868687a" path="/var/lib/kubelet/pods/86138fcf-3364-46db-ab0b-9c327868687a/volumes"
Oct 30 20:21:46.176754 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:46.176705    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"]
Oct 30 20:21:51.177290 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:51.177227    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j" podUID="d67582dc-0db8-4943-836f-fe3d44c53065" containerName="payload-pre-processing" containerID="cri-o://191c17a20b495b6d4a786f7195081383ccb03bb62725a2055b383ad3eb8fa9e7" gracePeriod=25
Oct 30 20:21:57.946790 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:21:57.946750    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j" podUID="d67582dc-0db8-4943-836f-fe3d44c53065" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:22:02.154021 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:02.153985    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"]
Oct 30 20:22:02.990209 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:02.990173    6030 scope.go:122] "RemoveContainer" containerID="ce3c52828a3f5dcbcfa5122852852de805238921c2e0dc1399526d3dfe53e751"
Oct 30 20:22:04.288407 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:22:04.288365    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:22:07.155072 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:07.155002    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb" podUID="35eee6aa-cd49-4b82-80d2-b680e545d1a4" containerName="payload-processing" containerID="cri-o://cb4cae5c7f74a6250ce7359f40260f5a191b793ead1e010071924262bfd2b69d" gracePeriod=25
Oct 30 20:22:07.315464 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:07.315429    6030 generic.go:356] "Generic (PLEG): container finished" podID="35eee6aa-cd49-4b82-80d2-b680e545d1a4" containerID="cb4cae5c7f74a6250ce7359f40260f5a191b793ead1e010071924262bfd2b69d" exitCode=0
Oct 30 20:22:07.315641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:07.315504    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb" event={"ID":"35eee6aa-cd49-4b82-80d2-b680e545d1a4","Type":"ContainerDied","Data":"cb4cae5c7f74a6250ce7359f40260f5a191b793ead1e010071924262bfd2b69d"}
Oct 30 20:22:07.415408 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:07.415352    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"
Oct 30 20:22:07.542309 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:07.542276    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35eee6aa-cd49-4b82-80d2-b680e545d1a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35eee6aa-cd49-4b82-80d2-b680e545d1a4-plugins-config-volume\") pod \"35eee6aa-cd49-4b82-80d2-b680e545d1a4\" (UID: \"35eee6aa-cd49-4b82-80d2-b680e545d1a4\") "
Oct 30 20:22:07.542487 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:07.542412    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35eee6aa-cd49-4b82-80d2-b680e545d1a4-kube-api-access-gj2kl\" (UniqueName: \"kubernetes.io/projected/35eee6aa-cd49-4b82-80d2-b680e545d1a4-kube-api-access-gj2kl\") pod \"35eee6aa-cd49-4b82-80d2-b680e545d1a4\" (UID: \"35eee6aa-cd49-4b82-80d2-b680e545d1a4\") "
Oct 30 20:22:07.542672 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:07.542649    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35eee6aa-cd49-4b82-80d2-b680e545d1a4-plugins-config-volume" pod "35eee6aa-cd49-4b82-80d2-b680e545d1a4" (UID: "35eee6aa-cd49-4b82-80d2-b680e545d1a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:07.544618 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:07.544594    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35eee6aa-cd49-4b82-80d2-b680e545d1a4-kube-api-access-gj2kl" pod "35eee6aa-cd49-4b82-80d2-b680e545d1a4" (UID: "35eee6aa-cd49-4b82-80d2-b680e545d1a4"). InnerVolumeSpecName "kube-api-access-gj2kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:07.643618 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:07.643581    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj2kl\" (UniqueName: \"kubernetes.io/projected/35eee6aa-cd49-4b82-80d2-b680e545d1a4-kube-api-access-gj2kl\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:07.643618 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:07.643614    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35eee6aa-cd49-4b82-80d2-b680e545d1a4-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:07.946378 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:07.946342    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j" podUID="d67582dc-0db8-4943-836f-fe3d44c53065" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:22:08.321700 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:08.321615    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb" event={"ID":"35eee6aa-cd49-4b82-80d2-b680e545d1a4","Type":"ContainerDied","Data":"0143c39675e17513ceaad17ac3882efe6fb718e059154613b453e5082c0dfa2e"}
Oct 30 20:22:08.321700 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:08.321681    6030 scope.go:122] "RemoveContainer" containerID="cb4cae5c7f74a6250ce7359f40260f5a191b793ead1e010071924262bfd2b69d"
Oct 30 20:22:08.322239 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:08.321625    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"
Oct 30 20:22:08.346832 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:08.346802    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"]
Oct 30 20:22:08.349560 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:08.349537    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5f9b3c-668c96494b-cg7kb"]
Oct 30 20:22:09.202426 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:09.202391    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35eee6aa-cd49-4b82-80d2-b680e545d1a4" path="/var/lib/kubelet/pods/35eee6aa-cd49-4b82-80d2-b680e545d1a4/volumes"
Oct 30 20:22:11.340191 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:11.340158    6030 generic.go:356] "Generic (PLEG): container finished" podID="d67582dc-0db8-4943-836f-fe3d44c53065" containerID="191c17a20b495b6d4a786f7195081383ccb03bb62725a2055b383ad3eb8fa9e7" exitCode=0
Oct 30 20:22:11.340663 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:11.340233    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j" event={"ID":"d67582dc-0db8-4943-836f-fe3d44c53065","Type":"ContainerDied","Data":"191c17a20b495b6d4a786f7195081383ccb03bb62725a2055b383ad3eb8fa9e7"}
Oct 30 20:22:11.475630 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:11.474680    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"
Oct 30 20:22:11.478801 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:11.477767    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67582dc-0db8-4943-836f-fe3d44c53065-kube-api-access-n6lhb\" (UniqueName: \"kubernetes.io/projected/d67582dc-0db8-4943-836f-fe3d44c53065-kube-api-access-n6lhb\") pod \"d67582dc-0db8-4943-836f-fe3d44c53065\" (UID: \"d67582dc-0db8-4943-836f-fe3d44c53065\") "
Oct 30 20:22:11.478801 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:11.477834    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d67582dc-0db8-4943-836f-fe3d44c53065-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67582dc-0db8-4943-836f-fe3d44c53065-plugins-config-volume\") pod \"d67582dc-0db8-4943-836f-fe3d44c53065\" (UID: \"d67582dc-0db8-4943-836f-fe3d44c53065\") "
Oct 30 20:22:11.479641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:11.479320    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67582dc-0db8-4943-836f-fe3d44c53065-plugins-config-volume" pod "d67582dc-0db8-4943-836f-fe3d44c53065" (UID: "d67582dc-0db8-4943-836f-fe3d44c53065"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:11.483947 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:11.483908    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67582dc-0db8-4943-836f-fe3d44c53065-kube-api-access-n6lhb" pod "d67582dc-0db8-4943-836f-fe3d44c53065" (UID: "d67582dc-0db8-4943-836f-fe3d44c53065"). InnerVolumeSpecName "kube-api-access-n6lhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:11.579575 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:11.579546    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6lhb\" (UniqueName: \"kubernetes.io/projected/d67582dc-0db8-4943-836f-fe3d44c53065-kube-api-access-n6lhb\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:11.579575 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:11.579575    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67582dc-0db8-4943-836f-fe3d44c53065-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:12.346766 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:12.346736    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"
Oct 30 20:22:12.347273 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:12.346738    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j" event={"ID":"d67582dc-0db8-4943-836f-fe3d44c53065","Type":"ContainerDied","Data":"b2a54e6ddc20dbed41a4c5d7564f69fad50d6dd378bf8e3c95cbff527cdc7724"}
Oct 30 20:22:12.347273 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:12.346872    6030 scope.go:122] "RemoveContainer" containerID="191c17a20b495b6d4a786f7195081383ccb03bb62725a2055b383ad3eb8fa9e7"
Oct 30 20:22:12.379228 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:12.379195    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"]
Oct 30 20:22:12.383355 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:12.383328    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bdf9e6-c87885855-7966j"]
Oct 30 20:22:13.199674 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:13.199646    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d67582dc-0db8-4943-836f-fe3d44c53065" path="/var/lib/kubelet/pods/d67582dc-0db8-4943-836f-fe3d44c53065/volumes"
Oct 30 20:22:18.509025 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:18.508987    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"]
Oct 30 20:22:18.509548 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:18.509305    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67" podUID="73a4f90b-70e2-4fae-86d2-0d27c0ef8d00" containerName="maas-api" containerID="cri-o://854688ef58f98ff8e6ff13355fb37496124161d86eb296dd733f837aa7560232" gracePeriod=30
Oct 30 20:22:18.813105 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:18.813078    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:22:18.882852 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:18.882819    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-maas-api-tls\") pod \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\" (UID: \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\") "
Oct 30 20:22:18.883023 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:18.882875    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-metrics-certs\") pod \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\" (UID: \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\") "
Oct 30 20:22:18.883023 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:18.882956    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-kube-api-access-stg9n\" (UniqueName: \"kubernetes.io/projected/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-kube-api-access-stg9n\") pod \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\" (UID: \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\") "
Oct 30 20:22:18.883191 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:18.883108    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-openshift-service-ca\") pod \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\" (UID: \"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00\") "
Oct 30 20:22:18.883469 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:18.883442    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-openshift-service-ca" pod "73a4f90b-70e2-4fae-86d2-0d27c0ef8d00" (UID: "73a4f90b-70e2-4fae-86d2-0d27c0ef8d00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:18.883777 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:18.883643    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:18.885227 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:18.885200    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-maas-api-tls" pod "73a4f90b-70e2-4fae-86d2-0d27c0ef8d00" (UID: "73a4f90b-70e2-4fae-86d2-0d27c0ef8d00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:18.885357 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:18.885226    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-kube-api-access-stg9n" pod "73a4f90b-70e2-4fae-86d2-0d27c0ef8d00" (UID: "73a4f90b-70e2-4fae-86d2-0d27c0ef8d00"). InnerVolumeSpecName "kube-api-access-stg9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:18.885357 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:18.885328    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-metrics-certs" pod "73a4f90b-70e2-4fae-86d2-0d27c0ef8d00" (UID: "73a4f90b-70e2-4fae-86d2-0d27c0ef8d00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:18.984874 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:18.984836    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stg9n\" (UniqueName: \"kubernetes.io/projected/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-kube-api-access-stg9n\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:18.984874 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:18.984868    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:18.984874 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:18.984878    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:19.389511 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:19.389476    6030 generic.go:356] "Generic (PLEG): container finished" podID="73a4f90b-70e2-4fae-86d2-0d27c0ef8d00" containerID="854688ef58f98ff8e6ff13355fb37496124161d86eb296dd733f837aa7560232" exitCode=0
Oct 30 20:22:19.389686 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:19.389528    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"
Oct 30 20:22:19.389686 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:19.389531    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67" event={"ID":"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00","Type":"ContainerDied","Data":"854688ef58f98ff8e6ff13355fb37496124161d86eb296dd733f837aa7560232"}
Oct 30 20:22:19.389686 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:19.389660    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67" event={"ID":"73a4f90b-70e2-4fae-86d2-0d27c0ef8d00","Type":"ContainerDied","Data":"20ad82e4e6a36f3bd675bc03388b93e8948388a893c20e0bcf60dcc43391f9ed"}
Oct 30 20:22:19.389845 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:19.389698    6030 scope.go:122] "RemoveContainer" containerID="854688ef58f98ff8e6ff13355fb37496124161d86eb296dd733f837aa7560232"
Oct 30 20:22:19.398876 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:19.398861    6030 scope.go:122] "RemoveContainer" containerID="854688ef58f98ff8e6ff13355fb37496124161d86eb296dd733f837aa7560232"
Oct 30 20:22:19.399174 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:22:19.399142    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854688ef58f98ff8e6ff13355fb37496124161d86eb296dd733f837aa7560232\": container with ID starting with 854688ef58f98ff8e6ff13355fb37496124161d86eb296dd733f837aa7560232 not found: ID does not exist" containerID="854688ef58f98ff8e6ff13355fb37496124161d86eb296dd733f837aa7560232"
Oct 30 20:22:19.399252 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:19.399182    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854688ef58f98ff8e6ff13355fb37496124161d86eb296dd733f837aa7560232"} err="failed to get container status \"854688ef58f98ff8e6ff13355fb37496124161d86eb296dd733f837aa7560232\": rpc error: code = NotFound desc = could not find container \"854688ef58f98ff8e6ff13355fb37496124161d86eb296dd733f837aa7560232\": container with ID starting with 854688ef58f98ff8e6ff13355fb37496124161d86eb296dd733f837aa7560232 not found: ID does not exist"
Oct 30 20:22:19.410206 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:19.410185    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"]
Oct 30 20:22:19.415944 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:19.415922    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-222480-85c747d68c-wgv67"]
Oct 30 20:22:21.200106 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:21.200069    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a4f90b-70e2-4fae-86d2-0d27c0ef8d00" path="/var/lib/kubelet/pods/73a4f90b-70e2-4fae-86d2-0d27c0ef8d00/volumes"
Oct 30 20:22:24.312435 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:24.312394    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"]
Oct 30 20:22:24.326629 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:24.326592    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"]
Oct 30 20:22:26.858991 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:26.858957    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"]
Oct 30 20:22:26.859512 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:26.859171    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg" podUID="c7bc2a39-e277-402d-9d0f-c71910d82058" containerName="maas-api" containerID="cri-o://97833b4a1d844b294a67ab8aec23f9a2fad45f9bacd18467112fe68a3ab17f3b" gracePeriod=30
Oct 30 20:22:27.161423 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.161395    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 20:22:27.270716 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.270672    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7bc2a39-e277-402d-9d0f-c71910d82058-kube-api-access-8w85n\" (UniqueName: \"kubernetes.io/projected/c7bc2a39-e277-402d-9d0f-c71910d82058-kube-api-access-8w85n\") pod \"c7bc2a39-e277-402d-9d0f-c71910d82058\" (UID: \"c7bc2a39-e277-402d-9d0f-c71910d82058\") "
Oct 30 20:22:27.270927 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.270746    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7bc2a39-e277-402d-9d0f-c71910d82058-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7bc2a39-e277-402d-9d0f-c71910d82058-openshift-service-ca\") pod \"c7bc2a39-e277-402d-9d0f-c71910d82058\" (UID: \"c7bc2a39-e277-402d-9d0f-c71910d82058\") "
Oct 30 20:22:27.270927 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.270875    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7bc2a39-e277-402d-9d0f-c71910d82058-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7bc2a39-e277-402d-9d0f-c71910d82058-maas-api-tls\") pod \"c7bc2a39-e277-402d-9d0f-c71910d82058\" (UID: \"c7bc2a39-e277-402d-9d0f-c71910d82058\") "
Oct 30 20:22:27.271132 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.270964    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7bc2a39-e277-402d-9d0f-c71910d82058-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7bc2a39-e277-402d-9d0f-c71910d82058-metrics-certs\") pod \"c7bc2a39-e277-402d-9d0f-c71910d82058\" (UID: \"c7bc2a39-e277-402d-9d0f-c71910d82058\") "
Oct 30 20:22:27.271431 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.271392    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7bc2a39-e277-402d-9d0f-c71910d82058-openshift-service-ca" pod "c7bc2a39-e277-402d-9d0f-c71910d82058" (UID: "c7bc2a39-e277-402d-9d0f-c71910d82058"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:27.273614 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.273456    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7bc2a39-e277-402d-9d0f-c71910d82058-kube-api-access-8w85n" pod "c7bc2a39-e277-402d-9d0f-c71910d82058" (UID: "c7bc2a39-e277-402d-9d0f-c71910d82058"). InnerVolumeSpecName "kube-api-access-8w85n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:27.274017 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.273977    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7bc2a39-e277-402d-9d0f-c71910d82058-maas-api-tls" pod "c7bc2a39-e277-402d-9d0f-c71910d82058" (UID: "c7bc2a39-e277-402d-9d0f-c71910d82058"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:27.274166 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.274105    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7bc2a39-e277-402d-9d0f-c71910d82058-metrics-certs" pod "c7bc2a39-e277-402d-9d0f-c71910d82058" (UID: "c7bc2a39-e277-402d-9d0f-c71910d82058"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:27.305086 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.304380    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"]
Oct 30 20:22:27.305086 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.304620    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8" podUID="5075d107-d1a3-4f56-afd0-6be45cbda01d" containerName="istio-proxy" containerID="cri-o://53b74d55c2fdeb0e080be3f797e078c087588b606c6b0a86069d67a497197f4a" gracePeriod=30
Oct 30 20:22:27.373141 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.373106    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7bc2a39-e277-402d-9d0f-c71910d82058-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:27.373141 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.373144    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w85n\" (UniqueName: \"kubernetes.io/projected/c7bc2a39-e277-402d-9d0f-c71910d82058-kube-api-access-8w85n\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:27.373361 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.373160    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7bc2a39-e277-402d-9d0f-c71910d82058-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:27.373361 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.373175    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7bc2a39-e277-402d-9d0f-c71910d82058-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:27.439622 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.439534    6030 generic.go:356] "Generic (PLEG): container finished" podID="c7bc2a39-e277-402d-9d0f-c71910d82058" containerID="97833b4a1d844b294a67ab8aec23f9a2fad45f9bacd18467112fe68a3ab17f3b" exitCode=0
Oct 30 20:22:27.439845 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.439588    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg" event={"ID":"c7bc2a39-e277-402d-9d0f-c71910d82058","Type":"ContainerDied","Data":"97833b4a1d844b294a67ab8aec23f9a2fad45f9bacd18467112fe68a3ab17f3b"}
Oct 30 20:22:27.439991 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.439609    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"
Oct 30 20:22:27.439991 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.439921    6030 scope.go:122] "RemoveContainer" containerID="97833b4a1d844b294a67ab8aec23f9a2fad45f9bacd18467112fe68a3ab17f3b"
Oct 30 20:22:27.440152 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.439904    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg" event={"ID":"c7bc2a39-e277-402d-9d0f-c71910d82058","Type":"ContainerDied","Data":"dd00af6d5f96afa7419a3f04280feedbf349932ec4cdd7a56dc1afbd240d3db7"}
Oct 30 20:22:27.452358 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.452337    6030 scope.go:122] "RemoveContainer" containerID="97833b4a1d844b294a67ab8aec23f9a2fad45f9bacd18467112fe68a3ab17f3b"
Oct 30 20:22:27.452673 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:22:27.452650    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97833b4a1d844b294a67ab8aec23f9a2fad45f9bacd18467112fe68a3ab17f3b\": container with ID starting with 97833b4a1d844b294a67ab8aec23f9a2fad45f9bacd18467112fe68a3ab17f3b not found: ID does not exist" containerID="97833b4a1d844b294a67ab8aec23f9a2fad45f9bacd18467112fe68a3ab17f3b"
Oct 30 20:22:27.452733 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.452688    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97833b4a1d844b294a67ab8aec23f9a2fad45f9bacd18467112fe68a3ab17f3b"} err="failed to get container status \"97833b4a1d844b294a67ab8aec23f9a2fad45f9bacd18467112fe68a3ab17f3b\": rpc error: code = NotFound desc = could not find container \"97833b4a1d844b294a67ab8aec23f9a2fad45f9bacd18467112fe68a3ab17f3b\": container with ID starting with 97833b4a1d844b294a67ab8aec23f9a2fad45f9bacd18467112fe68a3ab17f3b not found: ID does not exist"
Oct 30 20:22:27.471191 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.470710    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"]
Oct 30 20:22:27.472387 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:27.472362    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7eaeed-5c4f86dc57-wf7rg"]
Oct 30 20:22:29.199300 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:29.199257    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7bc2a39-e277-402d-9d0f-c71910d82058" path="/var/lib/kubelet/pods/c7bc2a39-e277-402d-9d0f-c71910d82058/volumes"
Oct 30 20:22:29.312990 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:29.312952    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p" podUID="4290bb83-6138-4d07-a193-f8fb4f5b2fb4" containerName="payload-processing" containerID="cri-o://964287b30e0cc22022fa2623262f532c4906acaa2527db2eae626b21a314e550" gracePeriod=25
Oct 30 20:22:29.327213 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:29.327166    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668" podUID="d043a6ac-9ac9-4f66-9669-e253eea8b381" containerName="payload-pre-processing" containerID="cri-o://cce1190d212eb6fc003eb3908ae34dc191bba302c35a0f2790b0be6ff491af7b" gracePeriod=25
Oct 30 20:22:32.468057 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.468008    6030 generic.go:356] "Generic (PLEG): container finished" podID="5075d107-d1a3-4f56-afd0-6be45cbda01d" containerID="53b74d55c2fdeb0e080be3f797e078c087588b606c6b0a86069d67a497197f4a" exitCode=0
Oct 30 20:22:32.468481 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.468081    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8" event={"ID":"5075d107-d1a3-4f56-afd0-6be45cbda01d","Type":"ContainerDied","Data":"53b74d55c2fdeb0e080be3f797e078c087588b606c6b0a86069d67a497197f4a"}
Oct 30 20:22:32.468481 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.468158    6030 scope.go:122] "RemoveContainer" containerID="28d5fb61c8d48c4c29739e388eee1b80641ecc70845eea1e282c0ff04af5b158"
Oct 30 20:22:32.558627 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.558595    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:22:32.629448 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.629422    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-credential-socket\") pod \"5075d107-d1a3-4f56-afd0-6be45cbda01d\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") "
Oct 30 20:22:32.629589 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.629462    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-data\") pod \"5075d107-d1a3-4f56-afd0-6be45cbda01d\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") "
Oct 30 20:22:32.629589 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.629510    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-workload-socket\") pod \"5075d107-d1a3-4f56-afd0-6be45cbda01d\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") "
Oct 30 20:22:32.629589 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.629528    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-token\" (UniqueName: \"kubernetes.io/projected/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-token\") pod \"5075d107-d1a3-4f56-afd0-6be45cbda01d\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") "
Oct 30 20:22:32.629764 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.629589    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5075d107-d1a3-4f56-afd0-6be45cbda01d-kube-api-access-rtqdx\" (UniqueName: \"kubernetes.io/projected/5075d107-d1a3-4f56-afd0-6be45cbda01d-kube-api-access-rtqdx\") pod \"5075d107-d1a3-4f56-afd0-6be45cbda01d\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") "
Oct 30 20:22:32.629764 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.629634    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-workload-certs\") pod \"5075d107-d1a3-4f56-afd0-6be45cbda01d\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") "
Oct 30 20:22:32.629764 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.629666    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-envoy\") pod \"5075d107-d1a3-4f56-afd0-6be45cbda01d\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") "
Oct 30 20:22:32.629764 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.629695    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-credential-socket" pod "5075d107-d1a3-4f56-afd0-6be45cbda01d" (UID: "5075d107-d1a3-4f56-afd0-6be45cbda01d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:32.630000 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.629751    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5075d107-d1a3-4f56-afd0-6be45cbda01d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5075d107-d1a3-4f56-afd0-6be45cbda01d-istiod-ca-cert\") pod \"5075d107-d1a3-4f56-afd0-6be45cbda01d\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") "
Oct 30 20:22:32.630000 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.629824    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-workload-socket" pod "5075d107-d1a3-4f56-afd0-6be45cbda01d" (UID: "5075d107-d1a3-4f56-afd0-6be45cbda01d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:32.630000 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.629841    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-data" pod "5075d107-d1a3-4f56-afd0-6be45cbda01d" (UID: "5075d107-d1a3-4f56-afd0-6be45cbda01d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:32.630000 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.629879    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-podinfo\") pod \"5075d107-d1a3-4f56-afd0-6be45cbda01d\" (UID: \"5075d107-d1a3-4f56-afd0-6be45cbda01d\") "
Oct 30 20:22:32.630220 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.629999    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-workload-certs" pod "5075d107-d1a3-4f56-afd0-6be45cbda01d" (UID: "5075d107-d1a3-4f56-afd0-6be45cbda01d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:32.630277 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.630238    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5075d107-d1a3-4f56-afd0-6be45cbda01d-istiod-ca-cert" pod "5075d107-d1a3-4f56-afd0-6be45cbda01d" (UID: "5075d107-d1a3-4f56-afd0-6be45cbda01d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:32.630403 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.630385    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:32.630453 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.630411    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:32.630453 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.630426    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:32.630453 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.630439    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:32.630601 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.630455    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5075d107-d1a3-4f56-afd0-6be45cbda01d-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:32.632118 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.632093    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-podinfo" pod "5075d107-d1a3-4f56-afd0-6be45cbda01d" (UID: "5075d107-d1a3-4f56-afd0-6be45cbda01d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:22:32.632286 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.632266    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5075d107-d1a3-4f56-afd0-6be45cbda01d-kube-api-access-rtqdx" pod "5075d107-d1a3-4f56-afd0-6be45cbda01d" (UID: "5075d107-d1a3-4f56-afd0-6be45cbda01d"). InnerVolumeSpecName "kube-api-access-rtqdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:32.632469 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.632451    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-token" pod "5075d107-d1a3-4f56-afd0-6be45cbda01d" (UID: "5075d107-d1a3-4f56-afd0-6be45cbda01d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:32.632546 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.632529    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-envoy" pod "5075d107-d1a3-4f56-afd0-6be45cbda01d" (UID: "5075d107-d1a3-4f56-afd0-6be45cbda01d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:32.731192 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.731157    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtqdx\" (UniqueName: \"kubernetes.io/projected/5075d107-d1a3-4f56-afd0-6be45cbda01d-kube-api-access-rtqdx\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:32.731192 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.731186    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:32.731192 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.731195    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:32.731427 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:32.731205    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5075d107-d1a3-4f56-afd0-6be45cbda01d-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:33.197238 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:33.197186    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p" podUID="4290bb83-6138-4d07-a193-f8fb4f5b2fb4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:22:33.197513 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:33.197255    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668" podUID="d043a6ac-9ac9-4f66-9669-e253eea8b381" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:22:33.476194 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:33.476110    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"
Oct 30 20:22:33.476194 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:33.476149    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8" event={"ID":"5075d107-d1a3-4f56-afd0-6be45cbda01d","Type":"ContainerDied","Data":"0fcb4aebe4d8118afabed93b1e41f22ee71a011b533486765f0bb60e20b5c8d1"}
Oct 30 20:22:33.476781 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:33.476212    6030 scope.go:122] "RemoveContainer" containerID="53b74d55c2fdeb0e080be3f797e078c087588b606c6b0a86069d67a497197f4a"
Oct 30 20:22:33.499761 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:33.499736    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"]
Oct 30 20:22:33.505780 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:33.505752    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-222480-openshift-default-69cb8bd57-z26b8"]
Oct 30 20:22:34.287357 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:22:34.287322    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:22:35.200895 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:35.200860    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5075d107-d1a3-4f56-afd0-6be45cbda01d" path="/var/lib/kubelet/pods/5075d107-d1a3-4f56-afd0-6be45cbda01d/volumes"
Oct 30 20:22:43.197406 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:43.197368    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p" podUID="4290bb83-6138-4d07-a193-f8fb4f5b2fb4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:22:43.197839 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:43.197368    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668" podUID="d043a6ac-9ac9-4f66-9669-e253eea8b381" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:22:45.869221 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:45.869183    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"]
Oct 30 20:22:49.562423 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.562385    6030 generic.go:356] "Generic (PLEG): container finished" podID="d043a6ac-9ac9-4f66-9669-e253eea8b381" containerID="cce1190d212eb6fc003eb3908ae34dc191bba302c35a0f2790b0be6ff491af7b" exitCode=0
Oct 30 20:22:49.562884 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.562509    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668" event={"ID":"d043a6ac-9ac9-4f66-9669-e253eea8b381","Type":"ContainerDied","Data":"cce1190d212eb6fc003eb3908ae34dc191bba302c35a0f2790b0be6ff491af7b"}
Oct 30 20:22:49.564299 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.564272    6030 generic.go:356] "Generic (PLEG): container finished" podID="4290bb83-6138-4d07-a193-f8fb4f5b2fb4" containerID="964287b30e0cc22022fa2623262f532c4906acaa2527db2eae626b21a314e550" exitCode=0
Oct 30 20:22:49.564406 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.564353    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p" event={"ID":"4290bb83-6138-4d07-a193-f8fb4f5b2fb4","Type":"ContainerDied","Data":"964287b30e0cc22022fa2623262f532c4906acaa2527db2eae626b21a314e550"}
Oct 30 20:22:49.564450 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.564412    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p" event={"ID":"4290bb83-6138-4d07-a193-f8fb4f5b2fb4","Type":"ContainerDied","Data":"6c66c34d5e61bc16da172846008f9fc44172a5c57a5787ec29843fedfcde3eba"}
Oct 30 20:22:49.564450 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.564436    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c66c34d5e61bc16da172846008f9fc44172a5c57a5787ec29843fedfcde3eba"
Oct 30 20:22:49.579396 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.579338    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"
Oct 30 20:22:49.603077 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.603038    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4290bb83-6138-4d07-a193-f8fb4f5b2fb4-kube-api-access-g5rlj\" (UniqueName: \"kubernetes.io/projected/4290bb83-6138-4d07-a193-f8fb4f5b2fb4-kube-api-access-g5rlj\") pod \"4290bb83-6138-4d07-a193-f8fb4f5b2fb4\" (UID: \"4290bb83-6138-4d07-a193-f8fb4f5b2fb4\") "
Oct 30 20:22:49.603245 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.603113    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4290bb83-6138-4d07-a193-f8fb4f5b2fb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4290bb83-6138-4d07-a193-f8fb4f5b2fb4-plugins-config-volume\") pod \"4290bb83-6138-4d07-a193-f8fb4f5b2fb4\" (UID: \"4290bb83-6138-4d07-a193-f8fb4f5b2fb4\") "
Oct 30 20:22:49.603519 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.603489    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4290bb83-6138-4d07-a193-f8fb4f5b2fb4-plugins-config-volume" pod "4290bb83-6138-4d07-a193-f8fb4f5b2fb4" (UID: "4290bb83-6138-4d07-a193-f8fb4f5b2fb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:49.605503 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.605474    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4290bb83-6138-4d07-a193-f8fb4f5b2fb4-kube-api-access-g5rlj" pod "4290bb83-6138-4d07-a193-f8fb4f5b2fb4" (UID: "4290bb83-6138-4d07-a193-f8fb4f5b2fb4"). InnerVolumeSpecName "kube-api-access-g5rlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:49.622180 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.622158    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"
Oct 30 20:22:49.703799 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.703773    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d043a6ac-9ac9-4f66-9669-e253eea8b381-kube-api-access-4t5v4\" (UniqueName: \"kubernetes.io/projected/d043a6ac-9ac9-4f66-9669-e253eea8b381-kube-api-access-4t5v4\") pod \"d043a6ac-9ac9-4f66-9669-e253eea8b381\" (UID: \"d043a6ac-9ac9-4f66-9669-e253eea8b381\") "
Oct 30 20:22:49.703958 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.703869    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d043a6ac-9ac9-4f66-9669-e253eea8b381-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d043a6ac-9ac9-4f66-9669-e253eea8b381-plugins-config-volume\") pod \"d043a6ac-9ac9-4f66-9669-e253eea8b381\" (UID: \"d043a6ac-9ac9-4f66-9669-e253eea8b381\") "
Oct 30 20:22:49.704170 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.704156    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5rlj\" (UniqueName: \"kubernetes.io/projected/4290bb83-6138-4d07-a193-f8fb4f5b2fb4-kube-api-access-g5rlj\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:49.704219 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.704173    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4290bb83-6138-4d07-a193-f8fb4f5b2fb4-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:49.704320 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.704293    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d043a6ac-9ac9-4f66-9669-e253eea8b381-plugins-config-volume" pod "d043a6ac-9ac9-4f66-9669-e253eea8b381" (UID: "d043a6ac-9ac9-4f66-9669-e253eea8b381"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:49.706009 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.705989    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d043a6ac-9ac9-4f66-9669-e253eea8b381-kube-api-access-4t5v4" pod "d043a6ac-9ac9-4f66-9669-e253eea8b381" (UID: "d043a6ac-9ac9-4f66-9669-e253eea8b381"). InnerVolumeSpecName "kube-api-access-4t5v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:49.805314 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.805283    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t5v4\" (UniqueName: \"kubernetes.io/projected/d043a6ac-9ac9-4f66-9669-e253eea8b381-kube-api-access-4t5v4\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:49.805314 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:49.805312    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d043a6ac-9ac9-4f66-9669-e253eea8b381-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:50.571523 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:50.571494    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"
Oct 30 20:22:50.572028 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:50.571515    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668" event={"ID":"d043a6ac-9ac9-4f66-9669-e253eea8b381","Type":"ContainerDied","Data":"0b58e50e74a9c7312f75ff8d39b6c50d641382f45addd56c404c7c726ff708d3"}
Oct 30 20:22:50.572028 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:50.571495    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"
Oct 30 20:22:50.572028 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:50.571616    6030 scope.go:122] "RemoveContainer" containerID="cce1190d212eb6fc003eb3908ae34dc191bba302c35a0f2790b0be6ff491af7b"
Oct 30 20:22:50.601181 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:50.601155    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"]
Oct 30 20:22:50.605697 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:50.605668    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7eaeed-55b5cfcb9b-8q54p"]
Oct 30 20:22:50.621809 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:50.621780    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"]
Oct 30 20:22:50.634925 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:50.634894    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7eaeed-787bbc65ff-rs668"]
Oct 30 20:22:50.869879 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:50.869776    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56" podUID="f9b8004a-deb0-418d-b6a9-bfd63122c373" containerName="payload-pre-processing" containerID="cri-o://d21c8794f47892ab48997aad0894b74024b3fcd9f98b1f1a68309eea2f6400ba" gracePeriod=25
Oct 30 20:22:51.202025 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:51.201985    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4290bb83-6138-4d07-a193-f8fb4f5b2fb4" path="/var/lib/kubelet/pods/4290bb83-6138-4d07-a193-f8fb4f5b2fb4/volumes"
Oct 30 20:22:51.202621 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:51.202600    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d043a6ac-9ac9-4f66-9669-e253eea8b381" path="/var/lib/kubelet/pods/d043a6ac-9ac9-4f66-9669-e253eea8b381/volumes"
Oct 30 20:22:58.783568 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:22:58.783527    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56" podUID="f9b8004a-deb0-418d-b6a9-bfd63122c373" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:23:02.130091 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:02.130035    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"]
Oct 30 20:23:02.135570 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:02.135540    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"]
Oct 30 20:23:02.177578 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:02.177545    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"]
Oct 30 20:23:03.035038 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:03.035006    6030 scope.go:122] "RemoveContainer" containerID="964287b30e0cc22022fa2623262f532c4906acaa2527db2eae626b21a314e550"
Oct 30 20:23:04.287226 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:23:04.287190    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:23:08.784485 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:08.784444    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56" podUID="f9b8004a-deb0-418d-b6a9-bfd63122c373" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:23:10.384258 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384216    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb"]
Oct 30 20:23:10.384703 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384688    6030 cpu_manager.go:414] "containerMap: removing container" podUID="fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0" containerName="payload-processing"
Oct 30 20:23:10.384753 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384705    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0" containerName="payload-processing"
Oct 30 20:23:10.384753 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384716    6030 cpu_manager.go:414] "containerMap: removing container" podUID="35eee6aa-cd49-4b82-80d2-b680e545d1a4" containerName="payload-processing"
Oct 30 20:23:10.384753 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384724    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35eee6aa-cd49-4b82-80d2-b680e545d1a4" containerName="payload-processing"
Oct 30 20:23:10.384753 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384745    6030 cpu_manager.go:414] "containerMap: removing container" podUID="f9310b6e-0c69-4e69-89a9-a268f132aa69" containerName="main"
Oct 30 20:23:10.384753 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384750    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9310b6e-0c69-4e69-89a9-a268f132aa69" containerName="main"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384757    6030 cpu_manager.go:414] "containerMap: removing container" podUID="d67582dc-0db8-4943-836f-fe3d44c53065" containerName="payload-pre-processing"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384762    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67582dc-0db8-4943-836f-fe3d44c53065" containerName="payload-pre-processing"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384770    6030 cpu_manager.go:414] "containerMap: removing container" podUID="5075d107-d1a3-4f56-afd0-6be45cbda01d" containerName="istio-proxy"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384774    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5075d107-d1a3-4f56-afd0-6be45cbda01d" containerName="istio-proxy"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384781    6030 cpu_manager.go:414] "containerMap: removing container" podUID="a083cbc5-d30b-4103-a5d8-a695e84418f2" containerName="payload-pre-processing"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384786    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a083cbc5-d30b-4103-a5d8-a695e84418f2" containerName="payload-pre-processing"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384793    6030 cpu_manager.go:414] "containerMap: removing container" podUID="5eb3ae64-0499-41d8-8f1b-f94f0ca43762" containerName="main"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384797    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb3ae64-0499-41d8-8f1b-f94f0ca43762" containerName="main"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384803    6030 cpu_manager.go:414] "containerMap: removing container" podUID="6484295e-f0bf-4194-9416-12c15129dd4a" containerName="istio-proxy"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384810    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6484295e-f0bf-4194-9416-12c15129dd4a" containerName="istio-proxy"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384822    6030 cpu_manager.go:414] "containerMap: removing container" podUID="914369a4-b049-4fa2-ab17-f31d0b21b083" containerName="payload-processing"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384827    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914369a4-b049-4fa2-ab17-f31d0b21b083" containerName="payload-processing"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384832    6030 cpu_manager.go:414] "containerMap: removing container" podUID="4290bb83-6138-4d07-a193-f8fb4f5b2fb4" containerName="payload-processing"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384836    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4290bb83-6138-4d07-a193-f8fb4f5b2fb4" containerName="payload-processing"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384850    6030 cpu_manager.go:414] "containerMap: removing container" podUID="73a4f90b-70e2-4fae-86d2-0d27c0ef8d00" containerName="maas-api"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384854    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a4f90b-70e2-4fae-86d2-0d27c0ef8d00" containerName="maas-api"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384861    6030 cpu_manager.go:414] "containerMap: removing container" podUID="6484295e-f0bf-4194-9416-12c15129dd4a" containerName="istio-proxy"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384865    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6484295e-f0bf-4194-9416-12c15129dd4a" containerName="istio-proxy"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384872    6030 cpu_manager.go:414] "containerMap: removing container" podUID="c7bc2a39-e277-402d-9d0f-c71910d82058" containerName="maas-api"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384875    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7bc2a39-e277-402d-9d0f-c71910d82058" containerName="maas-api"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384883    6030 cpu_manager.go:414] "containerMap: removing container" podUID="c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e" containerName="discovery"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384889    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e" containerName="discovery"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384906    6030 cpu_manager.go:414] "containerMap: removing container" podUID="86138fcf-3364-46db-ab0b-9c327868687a" containerName="maas-api"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384911    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86138fcf-3364-46db-ab0b-9c327868687a" containerName="maas-api"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384916    6030 cpu_manager.go:414] "containerMap: removing container" podUID="0f6f8eab-4484-4542-9e49-9b13bb4a8cef" containerName="maas-api"
Oct 30 20:23:10.384917 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384921    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f6f8eab-4484-4542-9e49-9b13bb4a8cef" containerName="maas-api"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384931    6030 cpu_manager.go:414] "containerMap: removing container" podUID="d043a6ac-9ac9-4f66-9669-e253eea8b381" containerName="payload-pre-processing"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.384936    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d043a6ac-9ac9-4f66-9669-e253eea8b381" containerName="payload-pre-processing"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385006    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9310b6e-0c69-4e69-89a9-a268f132aa69" containerName="main"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385015    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="d67582dc-0db8-4943-836f-fe3d44c53065" containerName="payload-pre-processing"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385021    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="5075d107-d1a3-4f56-afd0-6be45cbda01d" containerName="istio-proxy"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385026    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb3ae64-0499-41d8-8f1b-f94f0ca43762" containerName="main"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385031    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="a083cbc5-d30b-4103-a5d8-a695e84418f2" containerName="payload-pre-processing"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385036    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="6484295e-f0bf-4194-9416-12c15129dd4a" containerName="istio-proxy"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385057    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="4290bb83-6138-4d07-a193-f8fb4f5b2fb4" containerName="payload-processing"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385065    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="914369a4-b049-4fa2-ab17-f31d0b21b083" containerName="payload-processing"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385075    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a4f90b-70e2-4fae-86d2-0d27c0ef8d00" containerName="maas-api"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385081    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="6484295e-f0bf-4194-9416-12c15129dd4a" containerName="istio-proxy"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385085    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7bc2a39-e277-402d-9d0f-c71910d82058" containerName="maas-api"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385091    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d2f03f-1b37-4cca-94a1-2c434d9a8f1e" containerName="discovery"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385097    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="5075d107-d1a3-4f56-afd0-6be45cbda01d" containerName="istio-proxy"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385103    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="86138fcf-3364-46db-ab0b-9c327868687a" containerName="maas-api"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385108    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f6f8eab-4484-4542-9e49-9b13bb4a8cef" containerName="maas-api"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385114    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="d043a6ac-9ac9-4f66-9669-e253eea8b381" containerName="payload-pre-processing"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385120    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdc56e3-b6f7-4d5a-bdbd-6e7caca428a0" containerName="payload-processing"
Oct 30 20:23:10.385837 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.385124    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="35eee6aa-cd49-4b82-80d2-b680e545d1a4" containerName="payload-processing"
Oct 30 20:23:10.389899 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.389881    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb"
Oct 30 20:23:10.399614 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.399585    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb"]
Oct 30 20:23:10.482633 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.482599    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 20:23:10.482633 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.482623    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:10.531167 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.531135    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rg5k\" (UniqueName: \"kubernetes.io/projected/675d9d43-b63c-4eba-8792-034e78ab738e-kube-api-access-7rg5k\") pod \"maas-api-revoke-sub-0762643f-zlvtb\" (UID: \"675d9d43-b63c-4eba-8792-034e78ab738e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb"
Oct 30 20:23:10.531331 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.531256    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675d9d43-b63c-4eba-8792-034e78ab738e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0762643f-zlvtb\" (UID: \"675d9d43-b63c-4eba-8792-034e78ab738e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb"
Oct 30 20:23:10.531380 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.531353    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675d9d43-b63c-4eba-8792-034e78ab738e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0762643f-zlvtb\" (UID: \"675d9d43-b63c-4eba-8792-034e78ab738e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb"
Oct 30 20:23:10.632547 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.632429    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675d9d43-b63c-4eba-8792-034e78ab738e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0762643f-zlvtb\" (UID: \"675d9d43-b63c-4eba-8792-034e78ab738e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb"
Oct 30 20:23:10.632547 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.632511    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rg5k\" (UniqueName: \"kubernetes.io/projected/675d9d43-b63c-4eba-8792-034e78ab738e-kube-api-access-7rg5k\") pod \"maas-api-revoke-sub-0762643f-zlvtb\" (UID: \"675d9d43-b63c-4eba-8792-034e78ab738e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb"
Oct 30 20:23:10.632797 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.632578    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675d9d43-b63c-4eba-8792-034e78ab738e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0762643f-zlvtb\" (UID: \"675d9d43-b63c-4eba-8792-034e78ab738e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb"
Oct 30 20:23:10.633322 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.633292    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675d9d43-b63c-4eba-8792-034e78ab738e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0762643f-zlvtb\" (UID: \"675d9d43-b63c-4eba-8792-034e78ab738e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb"
Oct 30 20:23:10.635252 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.635195    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675d9d43-b63c-4eba-8792-034e78ab738e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0762643f-zlvtb\" (UID: \"675d9d43-b63c-4eba-8792-034e78ab738e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb"
Oct 30 20:23:10.645834 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.645791    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rg5k\" (UniqueName: \"kubernetes.io/projected/675d9d43-b63c-4eba-8792-034e78ab738e-kube-api-access-7rg5k\") pod \"maas-api-revoke-sub-0762643f-zlvtb\" (UID: \"675d9d43-b63c-4eba-8792-034e78ab738e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb"
Oct 30 20:23:10.715465 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.715436    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb"
Oct 30 20:23:10.994754 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:10.994724    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb"]
Oct 30 20:23:10.996766 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:23:10.996740    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675d9d43_b63c_4eba_8792_034e78ab738e.slice/crio-9747080f3b293f4888916d6e7456d720a0c6d9ff2c7585b06dc5202a7f366c39 WatchSource:0}: Error finding container 9747080f3b293f4888916d6e7456d720a0c6d9ff2c7585b06dc5202a7f366c39: Status 404 returned error can't find the container with id 9747080f3b293f4888916d6e7456d720a0c6d9ff2c7585b06dc5202a7f366c39
Oct 30 20:23:11.113776 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.113428    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"
Oct 30 20:23:11.138773 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.138685    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"]
Oct 30 20:23:11.138942 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.138894    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m" podUID="610afb5d-5aff-4fe9-bbbc-c8c82722b55d" containerName="istio-proxy" containerID="cri-o://14efaa9e6cb2697e102e7d46e40156edd0aa4339161db32a51e5fcbb4a5dda28" gracePeriod=30
Oct 30 20:23:11.239263 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.239218    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b8004a-deb0-418d-b6a9-bfd63122c373-kube-api-access-fwhtm\" (UniqueName: \"kubernetes.io/projected/f9b8004a-deb0-418d-b6a9-bfd63122c373-kube-api-access-fwhtm\") pod \"f9b8004a-deb0-418d-b6a9-bfd63122c373\" (UID: \"f9b8004a-deb0-418d-b6a9-bfd63122c373\") "
Oct 30 20:23:11.239480 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.239298    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b8004a-deb0-418d-b6a9-bfd63122c373-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b8004a-deb0-418d-b6a9-bfd63122c373-plugins-config-volume\") pod \"f9b8004a-deb0-418d-b6a9-bfd63122c373\" (UID: \"f9b8004a-deb0-418d-b6a9-bfd63122c373\") "
Oct 30 20:23:11.240142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.240114    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b8004a-deb0-418d-b6a9-bfd63122c373-plugins-config-volume" pod "f9b8004a-deb0-418d-b6a9-bfd63122c373" (UID: "f9b8004a-deb0-418d-b6a9-bfd63122c373"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:11.243555 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.243007    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b8004a-deb0-418d-b6a9-bfd63122c373-kube-api-access-fwhtm" pod "f9b8004a-deb0-418d-b6a9-bfd63122c373" (UID: "f9b8004a-deb0-418d-b6a9-bfd63122c373"). InnerVolumeSpecName "kube-api-access-fwhtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:11.340984 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.340959    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwhtm\" (UniqueName: \"kubernetes.io/projected/f9b8004a-deb0-418d-b6a9-bfd63122c373-kube-api-access-fwhtm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:11.340984 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.340983    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b8004a-deb0-418d-b6a9-bfd63122c373-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:11.691488 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.691452    6030 generic.go:356] "Generic (PLEG): container finished" podID="f9b8004a-deb0-418d-b6a9-bfd63122c373" containerID="d21c8794f47892ab48997aad0894b74024b3fcd9f98b1f1a68309eea2f6400ba" exitCode=0
Oct 30 20:23:11.691947 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.691499    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"
Oct 30 20:23:11.691947 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.691549    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56" event={"ID":"f9b8004a-deb0-418d-b6a9-bfd63122c373","Type":"ContainerDied","Data":"d21c8794f47892ab48997aad0894b74024b3fcd9f98b1f1a68309eea2f6400ba"}
Oct 30 20:23:11.691947 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.691614    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56" event={"ID":"f9b8004a-deb0-418d-b6a9-bfd63122c373","Type":"ContainerDied","Data":"7d6106d270ab25258bb586143ba6be5b8c1b3f1e47e983849ebe6d6f0a626246"}
Oct 30 20:23:11.691947 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.691646    6030 scope.go:122] "RemoveContainer" containerID="d21c8794f47892ab48997aad0894b74024b3fcd9f98b1f1a68309eea2f6400ba"
Oct 30 20:23:11.693588 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.693548    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb" event={"ID":"675d9d43-b63c-4eba-8792-034e78ab738e","Type":"ContainerStarted","Data":"c16a62524a8fb092f368e3867d6faf0e44ed65e014103314151d9e40c2a15e2e"}
Oct 30 20:23:11.693722 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.693597    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb" event={"ID":"675d9d43-b63c-4eba-8792-034e78ab738e","Type":"ContainerStarted","Data":"9747080f3b293f4888916d6e7456d720a0c6d9ff2c7585b06dc5202a7f366c39"}
Oct 30 20:23:11.702378 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.702363    6030 scope.go:122] "RemoveContainer" containerID="d21c8794f47892ab48997aad0894b74024b3fcd9f98b1f1a68309eea2f6400ba"
Oct 30 20:23:11.702675 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:23:11.702654    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21c8794f47892ab48997aad0894b74024b3fcd9f98b1f1a68309eea2f6400ba\": container with ID starting with d21c8794f47892ab48997aad0894b74024b3fcd9f98b1f1a68309eea2f6400ba not found: ID does not exist" containerID="d21c8794f47892ab48997aad0894b74024b3fcd9f98b1f1a68309eea2f6400ba"
Oct 30 20:23:11.702758 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.702688    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21c8794f47892ab48997aad0894b74024b3fcd9f98b1f1a68309eea2f6400ba"} err="failed to get container status \"d21c8794f47892ab48997aad0894b74024b3fcd9f98b1f1a68309eea2f6400ba\": rpc error: code = NotFound desc = could not find container \"d21c8794f47892ab48997aad0894b74024b3fcd9f98b1f1a68309eea2f6400ba\": container with ID starting with d21c8794f47892ab48997aad0894b74024b3fcd9f98b1f1a68309eea2f6400ba not found: ID does not exist"
Oct 30 20:23:11.725314 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.725256    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb" podStartSLOduration=1.72523844 podStartE2EDuration="1.72523844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:11.721535305 +0000 UTC m=+3070.637142162" watchObservedRunningTime="2026-10-30 20:23:11.72523844 +0000 UTC m=+3070.640845298"
Oct 30 20:23:11.738515 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.738486    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"]
Oct 30 20:23:11.743098 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:11.743074    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-981ee5-d656456c-vpm56"]
Oct 30 20:23:12.699344 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:12.699312    6030 generic.go:356] "Generic (PLEG): container finished" podID="675d9d43-b63c-4eba-8792-034e78ab738e" containerID="c16a62524a8fb092f368e3867d6faf0e44ed65e014103314151d9e40c2a15e2e" exitCode=0
Oct 30 20:23:12.699789 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:12.699399    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb" event={"ID":"675d9d43-b63c-4eba-8792-034e78ab738e","Type":"ContainerDied","Data":"c16a62524a8fb092f368e3867d6faf0e44ed65e014103314151d9e40c2a15e2e"}
Oct 30 20:23:13.199433 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:13.199388    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b8004a-deb0-418d-b6a9-bfd63122c373" path="/var/lib/kubelet/pods/f9b8004a-deb0-418d-b6a9-bfd63122c373/volumes"
Oct 30 20:23:13.846518 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:13.846489    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb"
Oct 30 20:23:13.968714 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:13.968627    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/675d9d43-b63c-4eba-8792-034e78ab738e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675d9d43-b63c-4eba-8792-034e78ab738e-maas-api-openshift-ca\") pod \"675d9d43-b63c-4eba-8792-034e78ab738e\" (UID: \"675d9d43-b63c-4eba-8792-034e78ab738e\") "
Oct 30 20:23:13.968714 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:13.968665    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/675d9d43-b63c-4eba-8792-034e78ab738e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675d9d43-b63c-4eba-8792-034e78ab738e-maas-api-xks-ca\") pod \"675d9d43-b63c-4eba-8792-034e78ab738e\" (UID: \"675d9d43-b63c-4eba-8792-034e78ab738e\") "
Oct 30 20:23:13.968714 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:13.968704    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/675d9d43-b63c-4eba-8792-034e78ab738e-kube-api-access-7rg5k\" (UniqueName: \"kubernetes.io/projected/675d9d43-b63c-4eba-8792-034e78ab738e-kube-api-access-7rg5k\") pod \"675d9d43-b63c-4eba-8792-034e78ab738e\" (UID: \"675d9d43-b63c-4eba-8792-034e78ab738e\") "
Oct 30 20:23:13.969028 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:13.968944    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675d9d43-b63c-4eba-8792-034e78ab738e-maas-api-openshift-ca" pod "675d9d43-b63c-4eba-8792-034e78ab738e" (UID: "675d9d43-b63c-4eba-8792-034e78ab738e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:13.970958 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:13.970934    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675d9d43-b63c-4eba-8792-034e78ab738e-kube-api-access-7rg5k" pod "675d9d43-b63c-4eba-8792-034e78ab738e" (UID: "675d9d43-b63c-4eba-8792-034e78ab738e"). InnerVolumeSpecName "kube-api-access-7rg5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:13.971107 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:13.970977    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/675d9d43-b63c-4eba-8792-034e78ab738e-maas-api-xks-ca" pod "675d9d43-b63c-4eba-8792-034e78ab738e" (UID: "675d9d43-b63c-4eba-8792-034e78ab738e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:14.070036 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:14.070006    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675d9d43-b63c-4eba-8792-034e78ab738e-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:14.070036 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:14.070036    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675d9d43-b63c-4eba-8792-034e78ab738e-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:14.070241 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:14.070066    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rg5k\" (UniqueName: \"kubernetes.io/projected/675d9d43-b63c-4eba-8792-034e78ab738e-kube-api-access-7rg5k\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:14.712096 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:14.712068    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb"
Oct 30 20:23:14.712268 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:14.712069    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0762643f-zlvtb" event={"ID":"675d9d43-b63c-4eba-8792-034e78ab738e","Type":"ContainerDied","Data":"9747080f3b293f4888916d6e7456d720a0c6d9ff2c7585b06dc5202a7f366c39"}
Oct 30 20:23:14.712268 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:14.712197    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9747080f3b293f4888916d6e7456d720a0c6d9ff2c7585b06dc5202a7f366c39"
Oct 30 20:23:16.401568 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.401534    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:23:16.593309 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.593240    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-envoy\") pod \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") "
Oct 30 20:23:16.593309 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.593290    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-kube-api-access-pglvt\" (UniqueName: \"kubernetes.io/projected/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-kube-api-access-pglvt\") pod \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") "
Oct 30 20:23:16.593561 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.593330    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-token\" (UniqueName: \"kubernetes.io/projected/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-token\") pod \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") "
Oct 30 20:23:16.593561 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.593355    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-credential-socket\") pod \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") "
Oct 30 20:23:16.593561 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.593393    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-workload-socket\") pod \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") "
Oct 30 20:23:16.593561 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.593422    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-podinfo\") pod \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") "
Oct 30 20:23:16.593561 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.593456    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-workload-certs\") pod \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") "
Oct 30 20:23:16.593825 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.593567    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-data\") pod \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") "
Oct 30 20:23:16.593825 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.593605    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istiod-ca-cert\") pod \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\" (UID: \"610afb5d-5aff-4fe9-bbbc-c8c82722b55d\") "
Oct 30 20:23:16.593825 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.593726    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-workload-socket" pod "610afb5d-5aff-4fe9-bbbc-c8c82722b55d" (UID: "610afb5d-5aff-4fe9-bbbc-c8c82722b55d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:16.593825 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.593740    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-credential-socket" pod "610afb5d-5aff-4fe9-bbbc-c8c82722b55d" (UID: "610afb5d-5aff-4fe9-bbbc-c8c82722b55d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:16.594089 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.593963    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-data" pod "610afb5d-5aff-4fe9-bbbc-c8c82722b55d" (UID: "610afb5d-5aff-4fe9-bbbc-c8c82722b55d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:16.594162 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.594087    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-workload-certs" pod "610afb5d-5aff-4fe9-bbbc-c8c82722b55d" (UID: "610afb5d-5aff-4fe9-bbbc-c8c82722b55d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:16.594162 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.594119    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istiod-ca-cert" pod "610afb5d-5aff-4fe9-bbbc-c8c82722b55d" (UID: "610afb5d-5aff-4fe9-bbbc-c8c82722b55d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:16.594259 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.594167    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:16.594259 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.594185    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:16.594259 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.594199    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:16.595754 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.595732    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-token" pod "610afb5d-5aff-4fe9-bbbc-c8c82722b55d" (UID: "610afb5d-5aff-4fe9-bbbc-c8c82722b55d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:16.596011 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.595985    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-podinfo" pod "610afb5d-5aff-4fe9-bbbc-c8c82722b55d" (UID: "610afb5d-5aff-4fe9-bbbc-c8c82722b55d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:16.596213 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.596193    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-envoy" pod "610afb5d-5aff-4fe9-bbbc-c8c82722b55d" (UID: "610afb5d-5aff-4fe9-bbbc-c8c82722b55d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:16.596274 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.596216    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-kube-api-access-pglvt" pod "610afb5d-5aff-4fe9-bbbc-c8c82722b55d" (UID: "610afb5d-5aff-4fe9-bbbc-c8c82722b55d"). InnerVolumeSpecName "kube-api-access-pglvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:16.695090 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.695032    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pglvt\" (UniqueName: \"kubernetes.io/projected/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-kube-api-access-pglvt\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:16.695090 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.695084    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:16.695090 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.695095    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:16.695309 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.695103    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:16.695309 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.695112    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:16.695309 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.695119    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/610afb5d-5aff-4fe9-bbbc-c8c82722b55d-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:16.722715 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.722677    6030 generic.go:356] "Generic (PLEG): container finished" podID="610afb5d-5aff-4fe9-bbbc-c8c82722b55d" containerID="14efaa9e6cb2697e102e7d46e40156edd0aa4339161db32a51e5fcbb4a5dda28" exitCode=0
Oct 30 20:23:16.722847 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.722737    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"
Oct 30 20:23:16.722847 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.722759    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m" event={"ID":"610afb5d-5aff-4fe9-bbbc-c8c82722b55d","Type":"ContainerDied","Data":"14efaa9e6cb2697e102e7d46e40156edd0aa4339161db32a51e5fcbb4a5dda28"}
Oct 30 20:23:16.722847 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.722817    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m" event={"ID":"610afb5d-5aff-4fe9-bbbc-c8c82722b55d","Type":"ContainerDied","Data":"4bd4b6ed28729ba2ea1898c471719a5043a64fb112fe528a2f0db4e492026799"}
Oct 30 20:23:16.722847 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.722841    6030 scope.go:122] "RemoveContainer" containerID="14efaa9e6cb2697e102e7d46e40156edd0aa4339161db32a51e5fcbb4a5dda28"
Oct 30 20:23:16.732624 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.732608    6030 scope.go:122] "RemoveContainer" containerID="14efaa9e6cb2697e102e7d46e40156edd0aa4339161db32a51e5fcbb4a5dda28"
Oct 30 20:23:16.732858 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:23:16.732834    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14efaa9e6cb2697e102e7d46e40156edd0aa4339161db32a51e5fcbb4a5dda28\": container with ID starting with 14efaa9e6cb2697e102e7d46e40156edd0aa4339161db32a51e5fcbb4a5dda28 not found: ID does not exist" containerID="14efaa9e6cb2697e102e7d46e40156edd0aa4339161db32a51e5fcbb4a5dda28"
Oct 30 20:23:16.732898 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.732874    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14efaa9e6cb2697e102e7d46e40156edd0aa4339161db32a51e5fcbb4a5dda28"} err="failed to get container status \"14efaa9e6cb2697e102e7d46e40156edd0aa4339161db32a51e5fcbb4a5dda28\": rpc error: code = NotFound desc = could not find container \"14efaa9e6cb2697e102e7d46e40156edd0aa4339161db32a51e5fcbb4a5dda28\": container with ID starting with 14efaa9e6cb2697e102e7d46e40156edd0aa4339161db32a51e5fcbb4a5dda28 not found: ID does not exist"
Oct 30 20:23:16.753332 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.753308    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"]
Oct 30 20:23:16.759748 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:16.759727    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-vz68m"]
Oct 30 20:23:17.186428 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.186361    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5" podUID="2db52234-3bbf-4a48-b905-dec230b1b3f8" containerName="main" containerID="cri-o://7843e1871620b86d08401fb524e26f52221f23c658cf0fd2dde10f459ae12008" gracePeriod=45
Oct 30 20:23:17.200977 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.200950    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="610afb5d-5aff-4fe9-bbbc-c8c82722b55d" path="/var/lib/kubelet/pods/610afb5d-5aff-4fe9-bbbc-c8c82722b55d/volumes"
Oct 30 20:23:17.202312 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.202264    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r" podUID="3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" containerName="main" containerID="cri-o://b1c8ae9aa337bc68cda478d3c298dc28ef5c7fbd13789f9b2604f083b7f82712" gracePeriod=45
Oct 30 20:23:17.211993 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.211965    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main" containerID="cri-o://3ec47560be88eb0e29f246ed59016796cc680a2ee5f9c503f3447c4eaf9d8f5c" gracePeriod=45
Oct 30 20:23:17.490074 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.490014    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 20:23:17.493951 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.493931    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 20:23:17.498927 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.498909    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 20:23:17.505322 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505304    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722a2823-1014-4139-8a7b-d6d6e70d1d2e-kube-api-access-qwhmw\" (UniqueName: \"kubernetes.io/projected/722a2823-1014-4139-8a7b-d6d6e70d1d2e-kube-api-access-qwhmw\") pod \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") "
Oct 30 20:23:17.505428 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505339    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-tmp-dir\") pod \"2db52234-3bbf-4a48-b905-dec230b1b3f8\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") "
Oct 30 20:23:17.505428 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505368    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/722a2823-1014-4139-8a7b-d6d6e70d1d2e-tls-certs\" (UniqueName: \"kubernetes.io/secret/722a2823-1014-4139-8a7b-d6d6e70d1d2e-tls-certs\") pod \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") "
Oct 30 20:23:17.505428 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505389    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-dshm\") pod \"2db52234-3bbf-4a48-b905-dec230b1b3f8\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") "
Oct 30 20:23:17.505428 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505407    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-home\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-home\") pod \"2db52234-3bbf-4a48-b905-dec230b1b3f8\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") "
Oct 30 20:23:17.505641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505437    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-tmp-dir\") pod \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") "
Oct 30 20:23:17.505641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505461    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-model-cache\") pod \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") "
Oct 30 20:23:17.505641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505480    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db52234-3bbf-4a48-b905-dec230b1b3f8-kube-api-access-48l9f\" (UniqueName: \"kubernetes.io/projected/2db52234-3bbf-4a48-b905-dec230b1b3f8-kube-api-access-48l9f\") pod \"2db52234-3bbf-4a48-b905-dec230b1b3f8\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") "
Oct 30 20:23:17.505641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505523    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-tmp-dir\") pod \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") "
Oct 30 20:23:17.505641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505555    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2db52234-3bbf-4a48-b905-dec230b1b3f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/2db52234-3bbf-4a48-b905-dec230b1b3f8-tls-certs\") pod \"2db52234-3bbf-4a48-b905-dec230b1b3f8\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") "
Oct 30 20:23:17.505641 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505606    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-dshm\") pod \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") "
Oct 30 20:23:17.505952 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505645    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-kube-api-access-dx49x\" (UniqueName: \"kubernetes.io/projected/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-kube-api-access-dx49x\") pod \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") "
Oct 30 20:23:17.505952 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505671    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-home\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-home\") pod \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\" (UID: \"722a2823-1014-4139-8a7b-d6d6e70d1d2e\") "
Oct 30 20:23:17.505952 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505649    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-tmp-dir" pod "2db52234-3bbf-4a48-b905-dec230b1b3f8" (UID: "2db52234-3bbf-4a48-b905-dec230b1b3f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:17.505952 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505680    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-tmp-dir" pod "3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" (UID: "3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:17.505952 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505709    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-dshm\") pod \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") "
Oct 30 20:23:17.505952 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505712    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-model-cache" pod "722a2823-1014-4139-8a7b-d6d6e70d1d2e" (UID: "722a2823-1014-4139-8a7b-d6d6e70d1d2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:17.505952 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505743    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-home\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-home\") pod \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") "
Oct 30 20:23:17.505952 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505776    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-home" pod "2db52234-3bbf-4a48-b905-dec230b1b3f8" (UID: "2db52234-3bbf-4a48-b905-dec230b1b3f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:17.505952 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505848    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-model-cache\") pod \"2db52234-3bbf-4a48-b905-dec230b1b3f8\" (UID: \"2db52234-3bbf-4a48-b905-dec230b1b3f8\") "
Oct 30 20:23:17.505952 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505877    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-model-cache\") pod \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") "
Oct 30 20:23:17.505952 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.505903    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-tls-certs\") pod \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\" (UID: \"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09\") "
Oct 30 20:23:17.506729 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.506123    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-tmp-dir" pod "722a2823-1014-4139-8a7b-d6d6e70d1d2e" (UID: "722a2823-1014-4139-8a7b-d6d6e70d1d2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:17.507195 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.507166    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-home" pod "3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" (UID: "3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:17.507377 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.507356    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-home" pod "722a2823-1014-4139-8a7b-d6d6e70d1d2e" (UID: "722a2823-1014-4139-8a7b-d6d6e70d1d2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:17.507875 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.507851    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-model-cache" pod "3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" (UID: "3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:17.508762 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.508735    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.508861 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.508766    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.508861 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.508833    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.508861 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.508856    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.509063 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.508869    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.509063 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.508881    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.509063 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.508892    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.509063 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.508904    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.509304 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.509186    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/722a2823-1014-4139-8a7b-d6d6e70d1d2e-tls-certs" pod "722a2823-1014-4139-8a7b-d6d6e70d1d2e" (UID: "722a2823-1014-4139-8a7b-d6d6e70d1d2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:17.509572 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.509493    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2db52234-3bbf-4a48-b905-dec230b1b3f8-tls-certs" pod "2db52234-3bbf-4a48-b905-dec230b1b3f8" (UID: "2db52234-3bbf-4a48-b905-dec230b1b3f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:17.509729 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.509705    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-model-cache" pod "2db52234-3bbf-4a48-b905-dec230b1b3f8" (UID: "2db52234-3bbf-4a48-b905-dec230b1b3f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:17.510151 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.510116    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-dshm" pod "3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" (UID: "3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:17.510151 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.510127    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722a2823-1014-4139-8a7b-d6d6e70d1d2e-kube-api-access-qwhmw" pod "722a2823-1014-4139-8a7b-d6d6e70d1d2e" (UID: "722a2823-1014-4139-8a7b-d6d6e70d1d2e"). InnerVolumeSpecName "kube-api-access-qwhmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:17.510511 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.510475    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-kube-api-access-dx49x" pod "3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" (UID: "3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09"). InnerVolumeSpecName "kube-api-access-dx49x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:17.510602 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.510532    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-dshm" pod "722a2823-1014-4139-8a7b-d6d6e70d1d2e" (UID: "722a2823-1014-4139-8a7b-d6d6e70d1d2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:17.510602 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.510539    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db52234-3bbf-4a48-b905-dec230b1b3f8-kube-api-access-48l9f" pod "2db52234-3bbf-4a48-b905-dec230b1b3f8" (UID: "2db52234-3bbf-4a48-b905-dec230b1b3f8"). InnerVolumeSpecName "kube-api-access-48l9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:17.510744 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.510714    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-tls-certs" pod "3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" (UID: "3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:17.511229 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.511208    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-dshm" pod "2db52234-3bbf-4a48-b905-dec230b1b3f8" (UID: "2db52234-3bbf-4a48-b905-dec230b1b3f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:17.610073 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.610017    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722a2823-1014-4139-8a7b-d6d6e70d1d2e-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.610073 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.610078    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx49x\" (UniqueName: \"kubernetes.io/projected/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-kube-api-access-dx49x\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.610276 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.610090    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.610276 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.610099    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.610276 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.610108    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.610276 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.610116    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwhmw\" (UniqueName: \"kubernetes.io/projected/722a2823-1014-4139-8a7b-d6d6e70d1d2e-kube-api-access-qwhmw\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.610276 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.610125    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722a2823-1014-4139-8a7b-d6d6e70d1d2e-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.610276 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.610132    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2db52234-3bbf-4a48-b905-dec230b1b3f8-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.610276 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.610140    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48l9f\" (UniqueName: \"kubernetes.io/projected/2db52234-3bbf-4a48-b905-dec230b1b3f8-kube-api-access-48l9f\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.610276 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.610148    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db52234-3bbf-4a48-b905-dec230b1b3f8-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:17.728171 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.728088    6030 generic.go:356] "Generic (PLEG): container finished" podID="2db52234-3bbf-4a48-b905-dec230b1b3f8" containerID="7843e1871620b86d08401fb524e26f52221f23c658cf0fd2dde10f459ae12008" exitCode=0
Oct 30 20:23:17.728171 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.728139    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"
Oct 30 20:23:17.728516 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.728170    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5" event={"ID":"2db52234-3bbf-4a48-b905-dec230b1b3f8","Type":"ContainerDied","Data":"7843e1871620b86d08401fb524e26f52221f23c658cf0fd2dde10f459ae12008"}
Oct 30 20:23:17.728516 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.728212    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5" event={"ID":"2db52234-3bbf-4a48-b905-dec230b1b3f8","Type":"ContainerDied","Data":"2c4e26d2273ed905ff14379c2e701e41a0dcf4f86cc29acd57d2e08a1a5c0a4f"}
Oct 30 20:23:17.728516 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.728226    6030 scope.go:122] "RemoveContainer" containerID="7843e1871620b86d08401fb524e26f52221f23c658cf0fd2dde10f459ae12008"
Oct 30 20:23:17.729555 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.729532    6030 generic.go:356] "Generic (PLEG): container finished" podID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerID="3ec47560be88eb0e29f246ed59016796cc680a2ee5f9c503f3447c4eaf9d8f5c" exitCode=0
Oct 30 20:23:17.729654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.729577    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"
Oct 30 20:23:17.729654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.729606    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" event={"ID":"722a2823-1014-4139-8a7b-d6d6e70d1d2e","Type":"ContainerDied","Data":"3ec47560be88eb0e29f246ed59016796cc680a2ee5f9c503f3447c4eaf9d8f5c"}
Oct 30 20:23:17.729654 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.729633    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf" event={"ID":"722a2823-1014-4139-8a7b-d6d6e70d1d2e","Type":"ContainerDied","Data":"a9f7ae371b7cc15687f730481d4c319e5e2db70bad9fcf1fc4c2f85aab6436a8"}
Oct 30 20:23:17.731081 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.731056    6030 generic.go:356] "Generic (PLEG): container finished" podID="3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" containerID="b1c8ae9aa337bc68cda478d3c298dc28ef5c7fbd13789f9b2604f083b7f82712" exitCode=0
Oct 30 20:23:17.731210 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.731111    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"
Oct 30 20:23:17.731210 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.731131    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r" event={"ID":"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09","Type":"ContainerDied","Data":"b1c8ae9aa337bc68cda478d3c298dc28ef5c7fbd13789f9b2604f083b7f82712"}
Oct 30 20:23:17.731210 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.731167    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r" event={"ID":"3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09","Type":"ContainerDied","Data":"36de1e800b906cfd4fc2f9eb07422181b6e4fd02b134969f5073386f0c3a5f3a"}
Oct 30 20:23:17.745937 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.745918    6030 scope.go:122] "RemoveContainer" containerID="7843e1871620b86d08401fb524e26f52221f23c658cf0fd2dde10f459ae12008"
Oct 30 20:23:17.746207 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:23:17.746190    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7843e1871620b86d08401fb524e26f52221f23c658cf0fd2dde10f459ae12008\": container with ID starting with 7843e1871620b86d08401fb524e26f52221f23c658cf0fd2dde10f459ae12008 not found: ID does not exist" containerID="7843e1871620b86d08401fb524e26f52221f23c658cf0fd2dde10f459ae12008"
Oct 30 20:23:17.746277 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.746214    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7843e1871620b86d08401fb524e26f52221f23c658cf0fd2dde10f459ae12008"} err="failed to get container status \"7843e1871620b86d08401fb524e26f52221f23c658cf0fd2dde10f459ae12008\": rpc error: code = NotFound desc = could not find container \"7843e1871620b86d08401fb524e26f52221f23c658cf0fd2dde10f459ae12008\": container with ID starting with 7843e1871620b86d08401fb524e26f52221f23c658cf0fd2dde10f459ae12008 not found: ID does not exist"
Oct 30 20:23:17.746277 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.746229    6030 scope.go:122] "RemoveContainer" containerID="3ec47560be88eb0e29f246ed59016796cc680a2ee5f9c503f3447c4eaf9d8f5c"
Oct 30 20:23:17.754564 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.754547    6030 scope.go:122] "RemoveContainer" containerID="3ec47560be88eb0e29f246ed59016796cc680a2ee5f9c503f3447c4eaf9d8f5c"
Oct 30 20:23:17.754832 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:23:17.754814    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ec47560be88eb0e29f246ed59016796cc680a2ee5f9c503f3447c4eaf9d8f5c\": container with ID starting with 3ec47560be88eb0e29f246ed59016796cc680a2ee5f9c503f3447c4eaf9d8f5c not found: ID does not exist" containerID="3ec47560be88eb0e29f246ed59016796cc680a2ee5f9c503f3447c4eaf9d8f5c"
Oct 30 20:23:17.754882 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.754848    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ec47560be88eb0e29f246ed59016796cc680a2ee5f9c503f3447c4eaf9d8f5c"} err="failed to get container status \"3ec47560be88eb0e29f246ed59016796cc680a2ee5f9c503f3447c4eaf9d8f5c\": rpc error: code = NotFound desc = could not find container \"3ec47560be88eb0e29f246ed59016796cc680a2ee5f9c503f3447c4eaf9d8f5c\": container with ID starting with 3ec47560be88eb0e29f246ed59016796cc680a2ee5f9c503f3447c4eaf9d8f5c not found: ID does not exist"
Oct 30 20:23:17.754882 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.754866    6030 scope.go:122] "RemoveContainer" containerID="b1c8ae9aa337bc68cda478d3c298dc28ef5c7fbd13789f9b2604f083b7f82712"
Oct 30 20:23:17.760956 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.760935    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"]
Oct 30 20:23:17.763869 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.763849    6030 scope.go:122] "RemoveContainer" containerID="b1c8ae9aa337bc68cda478d3c298dc28ef5c7fbd13789f9b2604f083b7f82712"
Oct 30 20:23:17.764141 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:23:17.764122    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c8ae9aa337bc68cda478d3c298dc28ef5c7fbd13789f9b2604f083b7f82712\": container with ID starting with b1c8ae9aa337bc68cda478d3c298dc28ef5c7fbd13789f9b2604f083b7f82712 not found: ID does not exist" containerID="b1c8ae9aa337bc68cda478d3c298dc28ef5c7fbd13789f9b2604f083b7f82712"
Oct 30 20:23:17.764201 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.764150    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c8ae9aa337bc68cda478d3c298dc28ef5c7fbd13789f9b2604f083b7f82712"} err="failed to get container status \"b1c8ae9aa337bc68cda478d3c298dc28ef5c7fbd13789f9b2604f083b7f82712\": rpc error: code = NotFound desc = could not find container \"b1c8ae9aa337bc68cda478d3c298dc28ef5c7fbd13789f9b2604f083b7f82712\": container with ID starting with b1c8ae9aa337bc68cda478d3c298dc28ef5c7fbd13789f9b2604f083b7f82712 not found: ID does not exist"
Oct 30 20:23:17.764201 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.764189    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/distinct-w1-7a745c-kserve-55676d64c-58bw5"]
Oct 30 20:23:17.777581 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.777562    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"]
Oct 30 20:23:17.785813 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.785792    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/distinct-2-w1-7a745c-kserve-55f8755d5b-gpq7r"]
Oct 30 20:23:17.805971 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.805951    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"]
Oct 30 20:23:17.811359 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:17.811321    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-7a745c/simulator-w1-7a745c-kserve-99c68d58f-ghctf"]
Oct 30 20:23:19.199522 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:19.199492    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2db52234-3bbf-4a48-b905-dec230b1b3f8" path="/var/lib/kubelet/pods/2db52234-3bbf-4a48-b905-dec230b1b3f8/volumes"
Oct 30 20:23:19.199947 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:19.199891    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" path="/var/lib/kubelet/pods/3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09/volumes"
Oct 30 20:23:19.200276 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:19.200263    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" path="/var/lib/kubelet/pods/722a2823-1014-4139-8a7b-d6d6e70d1d2e/volumes"
Oct 30 20:23:22.049756 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:22.049022    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"]
Oct 30 20:23:25.015640 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.015607    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"]
Oct 30 20:23:25.016148 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.015907    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d" podUID="5cff68cc-c745-41a6-b8ff-bc46fb7edcb3" containerName="maas-api" containerID="cri-o://2d124454ca590c1cc54dbbc98bea63c8def3859fdc9d675c6b2e124f4f046ddc" gracePeriod=30
Oct 30 20:23:25.307901 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.307872    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:23:25.385544 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.385513    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-maas-api-tls\") pod \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\" (UID: \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\") "
Oct 30 20:23:25.385544 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.385555    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-kube-api-access-ffrtg\" (UniqueName: \"kubernetes.io/projected/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-kube-api-access-ffrtg\") pod \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\" (UID: \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\") "
Oct 30 20:23:25.385785 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.385613    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-openshift-service-ca\") pod \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\" (UID: \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\") "
Oct 30 20:23:25.385785 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.385642    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-metrics-certs\") pod \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\" (UID: \"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3\") "
Oct 30 20:23:25.386023 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.385998    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-openshift-service-ca" pod "5cff68cc-c745-41a6-b8ff-bc46fb7edcb3" (UID: "5cff68cc-c745-41a6-b8ff-bc46fb7edcb3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:25.388117 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.388033    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-metrics-certs" pod "5cff68cc-c745-41a6-b8ff-bc46fb7edcb3" (UID: "5cff68cc-c745-41a6-b8ff-bc46fb7edcb3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:25.388117 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.388038    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-kube-api-access-ffrtg" pod "5cff68cc-c745-41a6-b8ff-bc46fb7edcb3" (UID: "5cff68cc-c745-41a6-b8ff-bc46fb7edcb3"). InnerVolumeSpecName "kube-api-access-ffrtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:25.388117 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.388099    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-maas-api-tls" pod "5cff68cc-c745-41a6-b8ff-bc46fb7edcb3" (UID: "5cff68cc-c745-41a6-b8ff-bc46fb7edcb3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:25.486438 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.486403    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-maas-api-tls\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:25.486438 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.486434    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffrtg\" (UniqueName: \"kubernetes.io/projected/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-kube-api-access-ffrtg\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:25.486438 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.486445    6030 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-openshift-service-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:25.486692 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.486454    6030 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3-metrics-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:25.776468 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.776433    6030 generic.go:356] "Generic (PLEG): container finished" podID="5cff68cc-c745-41a6-b8ff-bc46fb7edcb3" containerID="2d124454ca590c1cc54dbbc98bea63c8def3859fdc9d675c6b2e124f4f046ddc" exitCode=0
Oct 30 20:23:25.776625 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.776477    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"
Oct 30 20:23:25.776625 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.776521    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d" event={"ID":"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3","Type":"ContainerDied","Data":"2d124454ca590c1cc54dbbc98bea63c8def3859fdc9d675c6b2e124f4f046ddc"}
Oct 30 20:23:25.776625 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.776572    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d" event={"ID":"5cff68cc-c745-41a6-b8ff-bc46fb7edcb3","Type":"ContainerDied","Data":"9478157b757862571d9c40d39b01a2730110f5c322e0808f903796177c117717"}
Oct 30 20:23:25.776625 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.776593    6030 scope.go:122] "RemoveContainer" containerID="2d124454ca590c1cc54dbbc98bea63c8def3859fdc9d675c6b2e124f4f046ddc"
Oct 30 20:23:25.787002 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.786984    6030 scope.go:122] "RemoveContainer" containerID="2d124454ca590c1cc54dbbc98bea63c8def3859fdc9d675c6b2e124f4f046ddc"
Oct 30 20:23:25.787323 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:23:25.787302    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d124454ca590c1cc54dbbc98bea63c8def3859fdc9d675c6b2e124f4f046ddc\": container with ID starting with 2d124454ca590c1cc54dbbc98bea63c8def3859fdc9d675c6b2e124f4f046ddc not found: ID does not exist" containerID="2d124454ca590c1cc54dbbc98bea63c8def3859fdc9d675c6b2e124f4f046ddc"
Oct 30 20:23:25.787388 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.787332    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d124454ca590c1cc54dbbc98bea63c8def3859fdc9d675c6b2e124f4f046ddc"} err="failed to get container status \"2d124454ca590c1cc54dbbc98bea63c8def3859fdc9d675c6b2e124f4f046ddc\": rpc error: code = NotFound desc = could not find container \"2d124454ca590c1cc54dbbc98bea63c8def3859fdc9d675c6b2e124f4f046ddc\": container with ID starting with 2d124454ca590c1cc54dbbc98bea63c8def3859fdc9d675c6b2e124f4f046ddc not found: ID does not exist"
Oct 30 20:23:25.802934 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.802911    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"]
Oct 30 20:23:25.805029 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:25.805009    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75b361-69dbcb59cb-l272d"]
Oct 30 20:23:27.050132 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.050083    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc" podUID="ba53b339-9f04-4b57-b1f0-4ee28ba07af7" containerName="payload-pre-processing" containerID="cri-o://43d2634156394444edd7f85cebeaf4511a2bb0fc1a28dc5c7908a6e9f9ceda94" gracePeriod=25
Oct 30 20:23:27.200488 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.200454    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cff68cc-c745-41a6-b8ff-bc46fb7edcb3" path="/var/lib/kubelet/pods/5cff68cc-c745-41a6-b8ff-bc46fb7edcb3/volumes"
Oct 30 20:23:27.308293 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.308227    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"
Oct 30 20:23:27.405215 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.405184    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba53b339-9f04-4b57-b1f0-4ee28ba07af7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba53b339-9f04-4b57-b1f0-4ee28ba07af7-plugins-config-volume\") pod \"ba53b339-9f04-4b57-b1f0-4ee28ba07af7\" (UID: \"ba53b339-9f04-4b57-b1f0-4ee28ba07af7\") "
Oct 30 20:23:27.405401 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.405243    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba53b339-9f04-4b57-b1f0-4ee28ba07af7-kube-api-access-vfqpb\" (UniqueName: \"kubernetes.io/projected/ba53b339-9f04-4b57-b1f0-4ee28ba07af7-kube-api-access-vfqpb\") pod \"ba53b339-9f04-4b57-b1f0-4ee28ba07af7\" (UID: \"ba53b339-9f04-4b57-b1f0-4ee28ba07af7\") "
Oct 30 20:23:27.405597 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.405569    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba53b339-9f04-4b57-b1f0-4ee28ba07af7-plugins-config-volume" pod "ba53b339-9f04-4b57-b1f0-4ee28ba07af7" (UID: "ba53b339-9f04-4b57-b1f0-4ee28ba07af7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:27.407938 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.407763    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba53b339-9f04-4b57-b1f0-4ee28ba07af7-kube-api-access-vfqpb" pod "ba53b339-9f04-4b57-b1f0-4ee28ba07af7" (UID: "ba53b339-9f04-4b57-b1f0-4ee28ba07af7"). InnerVolumeSpecName "kube-api-access-vfqpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:27.506494 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.506461    6030 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba53b339-9f04-4b57-b1f0-4ee28ba07af7-plugins-config-volume\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:27.506494 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.506488    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfqpb\" (UniqueName: \"kubernetes.io/projected/ba53b339-9f04-4b57-b1f0-4ee28ba07af7-kube-api-access-vfqpb\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:27.788254 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.788215    6030 generic.go:356] "Generic (PLEG): container finished" podID="ba53b339-9f04-4b57-b1f0-4ee28ba07af7" containerID="43d2634156394444edd7f85cebeaf4511a2bb0fc1a28dc5c7908a6e9f9ceda94" exitCode=0
Oct 30 20:23:27.788446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.788266    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"
Oct 30 20:23:27.788446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.788310    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc" event={"ID":"ba53b339-9f04-4b57-b1f0-4ee28ba07af7","Type":"ContainerDied","Data":"43d2634156394444edd7f85cebeaf4511a2bb0fc1a28dc5c7908a6e9f9ceda94"}
Oct 30 20:23:27.788446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.788374    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc" event={"ID":"ba53b339-9f04-4b57-b1f0-4ee28ba07af7","Type":"ContainerDied","Data":"790f6a94e863f2d419be81bf3ce575547186f8c826c5ab03880c275757989252"}
Oct 30 20:23:27.788446 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.788406    6030 scope.go:122] "RemoveContainer" containerID="43d2634156394444edd7f85cebeaf4511a2bb0fc1a28dc5c7908a6e9f9ceda94"
Oct 30 20:23:27.798577 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.798559    6030 scope.go:122] "RemoveContainer" containerID="43d2634156394444edd7f85cebeaf4511a2bb0fc1a28dc5c7908a6e9f9ceda94"
Oct 30 20:23:27.798879 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:23:27.798857    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d2634156394444edd7f85cebeaf4511a2bb0fc1a28dc5c7908a6e9f9ceda94\": container with ID starting with 43d2634156394444edd7f85cebeaf4511a2bb0fc1a28dc5c7908a6e9f9ceda94 not found: ID does not exist" containerID="43d2634156394444edd7f85cebeaf4511a2bb0fc1a28dc5c7908a6e9f9ceda94"
Oct 30 20:23:27.798967 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.798887    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d2634156394444edd7f85cebeaf4511a2bb0fc1a28dc5c7908a6e9f9ceda94"} err="failed to get container status \"43d2634156394444edd7f85cebeaf4511a2bb0fc1a28dc5c7908a6e9f9ceda94\": rpc error: code = NotFound desc = could not find container \"43d2634156394444edd7f85cebeaf4511a2bb0fc1a28dc5c7908a6e9f9ceda94\": container with ID starting with 43d2634156394444edd7f85cebeaf4511a2bb0fc1a28dc5c7908a6e9f9ceda94 not found: ID does not exist"
Oct 30 20:23:27.816034 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.816007    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"]
Oct 30 20:23:27.822599 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:27.822576    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75b361-5865586895-n8bqc"]
Oct 30 20:23:29.198622 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:29.198587    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba53b339-9f04-4b57-b1f0-4ee28ba07af7" path="/var/lib/kubelet/pods/ba53b339-9f04-4b57-b1f0-4ee28ba07af7/volumes"
Oct 30 20:23:29.934932 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:29.933443    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"]
Oct 30 20:23:29.934932 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:29.933697    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr" podUID="cce705f6-f914-463d-9093-2599d0db1db7" containerName="istio-proxy" containerID="cri-o://5de93769953a93248b5d81d78718ca4cce25877baed6353d6995796e735f2e30" gracePeriod=30
Oct 30 20:23:33.939614 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.939578    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"]
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.939932    6030 cpu_manager.go:414] "containerMap: removing container" podUID="ba53b339-9f04-4b57-b1f0-4ee28ba07af7" containerName="payload-pre-processing"
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.939945    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba53b339-9f04-4b57-b1f0-4ee28ba07af7" containerName="payload-pre-processing"
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.939958    6030 cpu_manager.go:414] "containerMap: removing container" podUID="f9b8004a-deb0-418d-b6a9-bfd63122c373" containerName="payload-pre-processing"
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.939963    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b8004a-deb0-418d-b6a9-bfd63122c373" containerName="payload-pre-processing"
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.939975    6030 cpu_manager.go:414] "containerMap: removing container" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main"
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.939981    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main"
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.939988    6030 cpu_manager.go:414] "containerMap: removing container" podUID="3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" containerName="main"
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.939992    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" containerName="main"
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940005    6030 cpu_manager.go:414] "containerMap: removing container" podUID="675d9d43-b63c-4eba-8792-034e78ab738e" containerName="revoke-keys"
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940010    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="675d9d43-b63c-4eba-8792-034e78ab738e" containerName="revoke-keys"
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940019    6030 cpu_manager.go:414] "containerMap: removing container" podUID="5075d107-d1a3-4f56-afd0-6be45cbda01d" containerName="istio-proxy"
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940023    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5075d107-d1a3-4f56-afd0-6be45cbda01d" containerName="istio-proxy"
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940032    6030 cpu_manager.go:414] "containerMap: removing container" podUID="5cff68cc-c745-41a6-b8ff-bc46fb7edcb3" containerName="maas-api"
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940036    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cff68cc-c745-41a6-b8ff-bc46fb7edcb3" containerName="maas-api"
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940065    6030 cpu_manager.go:414] "containerMap: removing container" podUID="2db52234-3bbf-4a48-b905-dec230b1b3f8" containerName="main"
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940070    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db52234-3bbf-4a48-b905-dec230b1b3f8" containerName="main"
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940078    6030 cpu_manager.go:414] "containerMap: removing container" podUID="610afb5d-5aff-4fe9-bbbc-c8c82722b55d" containerName="istio-proxy"
Oct 30 20:23:33.940088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940082    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610afb5d-5aff-4fe9-bbbc-c8c82722b55d" containerName="istio-proxy"
Oct 30 20:23:33.940758 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940134    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cff68cc-c745-41a6-b8ff-bc46fb7edcb3" containerName="maas-api"
Oct 30 20:23:33.940758 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940144    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db52234-3bbf-4a48-b905-dec230b1b3f8" containerName="main"
Oct 30 20:23:33.940758 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940150    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="610afb5d-5aff-4fe9-bbbc-c8c82722b55d" containerName="istio-proxy"
Oct 30 20:23:33.940758 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940156    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba53b339-9f04-4b57-b1f0-4ee28ba07af7" containerName="payload-pre-processing"
Oct 30 20:23:33.940758 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940161    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b8004a-deb0-418d-b6a9-bfd63122c373" containerName="payload-pre-processing"
Oct 30 20:23:33.940758 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940167    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="722a2823-1014-4139-8a7b-d6d6e70d1d2e" containerName="main"
Oct 30 20:23:33.940758 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940171    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1c5f0b-c1e2-4b93-b2d4-be4ece056a09" containerName="main"
Oct 30 20:23:33.940758 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.940178    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="675d9d43-b63c-4eba-8792-034e78ab738e" containerName="revoke-keys"
Oct 30 20:23:33.942657 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.942640    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"
Oct 30 20:23:33.953088 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.953060    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"]
Oct 30 20:23:33.953329 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.953303    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:23:33.953433 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:33.953379    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:23:34.010859 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:34.010814    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:34.011017 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:34.010977    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 20:23:34.067950 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:34.067923    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af782ac5-c43b-4ff0-994d-0a2f688ef479-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n\" (UID: \"af782ac5-c43b-4ff0-994d-0a2f688ef479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"
Oct 30 20:23:34.068135 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:34.067964    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gjgw\" (UniqueName: \"kubernetes.io/projected/af782ac5-c43b-4ff0-994d-0a2f688ef479-kube-api-access-4gjgw\") pod \"maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n\" (UID: \"af782ac5-c43b-4ff0-994d-0a2f688ef479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"
Oct 30 20:23:34.068135 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:34.068037    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af782ac5-c43b-4ff0-994d-0a2f688ef479-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n\" (UID: \"af782ac5-c43b-4ff0-994d-0a2f688ef479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"
Oct 30 20:23:34.169444 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:34.169407    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af782ac5-c43b-4ff0-994d-0a2f688ef479-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n\" (UID: \"af782ac5-c43b-4ff0-994d-0a2f688ef479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"
Oct 30 20:23:34.169609 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:34.169458    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gjgw\" (UniqueName: \"kubernetes.io/projected/af782ac5-c43b-4ff0-994d-0a2f688ef479-kube-api-access-4gjgw\") pod \"maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n\" (UID: \"af782ac5-c43b-4ff0-994d-0a2f688ef479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"
Oct 30 20:23:34.169609 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:34.169507    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af782ac5-c43b-4ff0-994d-0a2f688ef479-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n\" (UID: \"af782ac5-c43b-4ff0-994d-0a2f688ef479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"
Oct 30 20:23:34.170243 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:34.170218    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af782ac5-c43b-4ff0-994d-0a2f688ef479-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n\" (UID: \"af782ac5-c43b-4ff0-994d-0a2f688ef479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"
Oct 30 20:23:34.172001 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:34.171976    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af782ac5-c43b-4ff0-994d-0a2f688ef479-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n\" (UID: \"af782ac5-c43b-4ff0-994d-0a2f688ef479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"
Oct 30 20:23:34.183181 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:34.183159    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gjgw\" (UniqueName: \"kubernetes.io/projected/af782ac5-c43b-4ff0-994d-0a2f688ef479-kube-api-access-4gjgw\") pod \"maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n\" (UID: \"af782ac5-c43b-4ff0-994d-0a2f688ef479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"
Oct 30 20:23:34.256412 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:34.256337    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"
Oct 30 20:23:34.292300 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:23:34.292264    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:23:34.537565 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:34.537542    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"]
Oct 30 20:23:34.540368 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:23:34.540342    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf782ac5_c43b_4ff0_994d_0a2f688ef479.slice/crio-266fa3a26b69709c4e97bee11f705fe9617ebe5ec0fe262ec2f268d74e4a2770 WatchSource:0}: Error finding container 266fa3a26b69709c4e97bee11f705fe9617ebe5ec0fe262ec2f268d74e4a2770: Status 404 returned error can't find the container with id 266fa3a26b69709c4e97bee11f705fe9617ebe5ec0fe262ec2f268d74e4a2770
Oct 30 20:23:34.822373 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:34.822307    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n" event={"ID":"af782ac5-c43b-4ff0-994d-0a2f688ef479","Type":"ContainerStarted","Data":"01c1fe34f205b1618c37ff43041b530d54f3cef3beece6a4e939da3548ae8b45"}
Oct 30 20:23:34.822493 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:34.822390    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n" event={"ID":"af782ac5-c43b-4ff0-994d-0a2f688ef479","Type":"ContainerStarted","Data":"266fa3a26b69709c4e97bee11f705fe9617ebe5ec0fe262ec2f268d74e4a2770"}
Oct 30 20:23:34.846098 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:34.844504    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n" podStartSLOduration=1.844486956 podStartE2EDuration="1.844486956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:34.843169488 +0000 UTC m=+3093.758776344" watchObservedRunningTime="2026-10-30 20:23:34.844486956 +0000 UTC m=+3093.760093813"
Oct 30 20:23:35.173701 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.173675    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 20:23:35.279032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.279001    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-credential-socket\") pod \"cce705f6-f914-463d-9093-2599d0db1db7\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") "
Oct 30 20:23:35.279032 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.279061    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-workload-certs\") pod \"cce705f6-f914-463d-9093-2599d0db1db7\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") "
Oct 30 20:23:35.279320 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.279088    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-workload-socket\") pod \"cce705f6-f914-463d-9093-2599d0db1db7\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") "
Oct 30 20:23:35.279320 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.279125    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce705f6-f914-463d-9093-2599d0db1db7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce705f6-f914-463d-9093-2599d0db1db7-istiod-ca-cert\") pod \"cce705f6-f914-463d-9093-2599d0db1db7\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") "
Oct 30 20:23:35.279320 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.279157    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-istio-envoy\") pod \"cce705f6-f914-463d-9093-2599d0db1db7\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") "
Oct 30 20:23:35.279320 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.279210    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce705f6-f914-463d-9093-2599d0db1db7-istio-token\" (UniqueName: \"kubernetes.io/projected/cce705f6-f914-463d-9093-2599d0db1db7-istio-token\") pod \"cce705f6-f914-463d-9093-2599d0db1db7\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") "
Oct 30 20:23:35.279320 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.279285    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-istio-data\") pod \"cce705f6-f914-463d-9093-2599d0db1db7\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") "
Oct 30 20:23:35.279320 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.279281    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-credential-socket" pod "cce705f6-f914-463d-9093-2599d0db1db7" (UID: "cce705f6-f914-463d-9093-2599d0db1db7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:35.279320 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.279322    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce705f6-f914-463d-9093-2599d0db1db7-kube-api-access-lx459\" (UniqueName: \"kubernetes.io/projected/cce705f6-f914-463d-9093-2599d0db1db7-kube-api-access-lx459\") pod \"cce705f6-f914-463d-9093-2599d0db1db7\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") "
Oct 30 20:23:35.279715 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.279363    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cce705f6-f914-463d-9093-2599d0db1db7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce705f6-f914-463d-9093-2599d0db1db7-istio-podinfo\") pod \"cce705f6-f914-463d-9093-2599d0db1db7\" (UID: \"cce705f6-f914-463d-9093-2599d0db1db7\") "
Oct 30 20:23:35.279715 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.279362    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-workload-certs" pod "cce705f6-f914-463d-9093-2599d0db1db7" (UID: "cce705f6-f914-463d-9093-2599d0db1db7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:35.279715 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.279485    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-workload-socket" pod "cce705f6-f914-463d-9093-2599d0db1db7" (UID: "cce705f6-f914-463d-9093-2599d0db1db7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:35.279715 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.279656    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.279715 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.279693    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.279715 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.279707    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.279715 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.279652    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce705f6-f914-463d-9093-2599d0db1db7-istiod-ca-cert" pod "cce705f6-f914-463d-9093-2599d0db1db7" (UID: "cce705f6-f914-463d-9093-2599d0db1db7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:35.280064 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.279829    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-istio-data" pod "cce705f6-f914-463d-9093-2599d0db1db7" (UID: "cce705f6-f914-463d-9093-2599d0db1db7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:35.281581 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.281561    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cce705f6-f914-463d-9093-2599d0db1db7-istio-podinfo" pod "cce705f6-f914-463d-9093-2599d0db1db7" (UID: "cce705f6-f914-463d-9093-2599d0db1db7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:35.281690 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.281604    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-istio-envoy" pod "cce705f6-f914-463d-9093-2599d0db1db7" (UID: "cce705f6-f914-463d-9093-2599d0db1db7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:35.282143 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.282116    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce705f6-f914-463d-9093-2599d0db1db7-kube-api-access-lx459" pod "cce705f6-f914-463d-9093-2599d0db1db7" (UID: "cce705f6-f914-463d-9093-2599d0db1db7"). InnerVolumeSpecName "kube-api-access-lx459". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:35.282143 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.282122    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce705f6-f914-463d-9093-2599d0db1db7-istio-token" pod "cce705f6-f914-463d-9093-2599d0db1db7" (UID: "cce705f6-f914-463d-9093-2599d0db1db7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:35.381055 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.381023    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.381204 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.381068    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx459\" (UniqueName: \"kubernetes.io/projected/cce705f6-f914-463d-9093-2599d0db1db7-kube-api-access-lx459\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.381204 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.381083    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce705f6-f914-463d-9093-2599d0db1db7-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.381204 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.381093    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce705f6-f914-463d-9093-2599d0db1db7-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.381204 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.381102    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce705f6-f914-463d-9093-2599d0db1db7-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.381204 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.381110    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce705f6-f914-463d-9093-2599d0db1db7-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.827872 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.827778    6030 generic.go:356] "Generic (PLEG): container finished" podID="cce705f6-f914-463d-9093-2599d0db1db7" containerID="5de93769953a93248b5d81d78718ca4cce25877baed6353d6995796e735f2e30" exitCode=0
Oct 30 20:23:35.827872 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.827832    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"
Oct 30 20:23:35.827872 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.827857    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr" event={"ID":"cce705f6-f914-463d-9093-2599d0db1db7","Type":"ContainerDied","Data":"5de93769953a93248b5d81d78718ca4cce25877baed6353d6995796e735f2e30"}
Oct 30 20:23:35.828208 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.827907    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr" event={"ID":"cce705f6-f914-463d-9093-2599d0db1db7","Type":"ContainerDied","Data":"e8bbe82656796da1ba300278be2cbd07073820f485bfd3d334e1f9b29b8d8a7b"}
Oct 30 20:23:35.828208 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.827933    6030 scope.go:122] "RemoveContainer" containerID="5de93769953a93248b5d81d78718ca4cce25877baed6353d6995796e735f2e30"
Oct 30 20:23:35.829483 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.829463    6030 generic.go:356] "Generic (PLEG): container finished" podID="af782ac5-c43b-4ff0-994d-0a2f688ef479" containerID="01c1fe34f205b1618c37ff43041b530d54f3cef3beece6a4e939da3548ae8b45" exitCode=0
Oct 30 20:23:35.829675 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.829554    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n" event={"ID":"af782ac5-c43b-4ff0-994d-0a2f688ef479","Type":"ContainerDied","Data":"01c1fe34f205b1618c37ff43041b530d54f3cef3beece6a4e939da3548ae8b45"}
Oct 30 20:23:35.837358 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.837337    6030 scope.go:122] "RemoveContainer" containerID="5de93769953a93248b5d81d78718ca4cce25877baed6353d6995796e735f2e30"
Oct 30 20:23:35.837677 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:23:35.837652    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de93769953a93248b5d81d78718ca4cce25877baed6353d6995796e735f2e30\": container with ID starting with 5de93769953a93248b5d81d78718ca4cce25877baed6353d6995796e735f2e30 not found: ID does not exist" containerID="5de93769953a93248b5d81d78718ca4cce25877baed6353d6995796e735f2e30"
Oct 30 20:23:35.837761 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.837692    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de93769953a93248b5d81d78718ca4cce25877baed6353d6995796e735f2e30"} err="failed to get container status \"5de93769953a93248b5d81d78718ca4cce25877baed6353d6995796e735f2e30\": rpc error: code = NotFound desc = could not find container \"5de93769953a93248b5d81d78718ca4cce25877baed6353d6995796e735f2e30\": container with ID starting with 5de93769953a93248b5d81d78718ca4cce25877baed6353d6995796e735f2e30 not found: ID does not exist"
Oct 30 20:23:35.880736 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.880704    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"]
Oct 30 20:23:35.884308 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:35.884281    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-885fr"]
Oct 30 20:23:36.966663 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:36.966641    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"
Oct 30 20:23:37.098938 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:37.098858    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af782ac5-c43b-4ff0-994d-0a2f688ef479-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af782ac5-c43b-4ff0-994d-0a2f688ef479-maas-api-openshift-ca\") pod \"af782ac5-c43b-4ff0-994d-0a2f688ef479\" (UID: \"af782ac5-c43b-4ff0-994d-0a2f688ef479\") "
Oct 30 20:23:37.098938 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:37.098930    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af782ac5-c43b-4ff0-994d-0a2f688ef479-kube-api-access-4gjgw\" (UniqueName: \"kubernetes.io/projected/af782ac5-c43b-4ff0-994d-0a2f688ef479-kube-api-access-4gjgw\") pod \"af782ac5-c43b-4ff0-994d-0a2f688ef479\" (UID: \"af782ac5-c43b-4ff0-994d-0a2f688ef479\") "
Oct 30 20:23:37.099175 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:37.099065    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af782ac5-c43b-4ff0-994d-0a2f688ef479-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af782ac5-c43b-4ff0-994d-0a2f688ef479-maas-api-xks-ca\") pod \"af782ac5-c43b-4ff0-994d-0a2f688ef479\" (UID: \"af782ac5-c43b-4ff0-994d-0a2f688ef479\") "
Oct 30 20:23:37.099294 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:37.099259    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af782ac5-c43b-4ff0-994d-0a2f688ef479-maas-api-openshift-ca" pod "af782ac5-c43b-4ff0-994d-0a2f688ef479" (UID: "af782ac5-c43b-4ff0-994d-0a2f688ef479"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:37.101209 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:37.101183    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af782ac5-c43b-4ff0-994d-0a2f688ef479-maas-api-xks-ca" pod "af782ac5-c43b-4ff0-994d-0a2f688ef479" (UID: "af782ac5-c43b-4ff0-994d-0a2f688ef479"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:37.101344 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:37.101321    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af782ac5-c43b-4ff0-994d-0a2f688ef479-kube-api-access-4gjgw" pod "af782ac5-c43b-4ff0-994d-0a2f688ef479" (UID: "af782ac5-c43b-4ff0-994d-0a2f688ef479"). InnerVolumeSpecName "kube-api-access-4gjgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:37.199438 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:37.199405    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce705f6-f914-463d-9093-2599d0db1db7" path="/var/lib/kubelet/pods/cce705f6-f914-463d-9093-2599d0db1db7/volumes"
Oct 30 20:23:37.199680 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:37.199662    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af782ac5-c43b-4ff0-994d-0a2f688ef479-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:37.199728 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:37.199684    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af782ac5-c43b-4ff0-994d-0a2f688ef479-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:37.199769 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:37.199749    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gjgw\" (UniqueName: \"kubernetes.io/projected/af782ac5-c43b-4ff0-994d-0a2f688ef479-kube-api-access-4gjgw\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:37.843653 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:37.843622    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"
Oct 30 20:23:37.843841 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:37.843656    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n" event={"ID":"af782ac5-c43b-4ff0-994d-0a2f688ef479","Type":"ContainerDied","Data":"266fa3a26b69709c4e97bee11f705fe9617ebe5ec0fe262ec2f268d74e4a2770"}
Oct 30 20:23:37.843841 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:37.843699    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="266fa3a26b69709c4e97bee11f705fe9617ebe5ec0fe262ec2f268d74e4a2770"
Oct 30 20:23:38.776649 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:38.776613    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"]
Oct 30 20:23:38.789027 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:38.788997    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"]
Oct 30 20:23:38.804969 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:38.804939    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"]
Oct 30 20:23:44.893040 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:44.893006    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659"]
Oct 30 20:23:44.893582 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:44.893370    6030 cpu_manager.go:414] "containerMap: removing container" podUID="cce705f6-f914-463d-9093-2599d0db1db7" containerName="istio-proxy"
Oct 30 20:23:44.893582 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:44.893380    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce705f6-f914-463d-9093-2599d0db1db7" containerName="istio-proxy"
Oct 30 20:23:44.893582 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:44.893403    6030 cpu_manager.go:414] "containerMap: removing container" podUID="af782ac5-c43b-4ff0-994d-0a2f688ef479" containerName="revoke-keys"
Oct 30 20:23:44.893582 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:44.893408    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af782ac5-c43b-4ff0-994d-0a2f688ef479" containerName="revoke-keys"
Oct 30 20:23:44.893582 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:44.893464    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce705f6-f914-463d-9093-2599d0db1db7" containerName="istio-proxy"
Oct 30 20:23:44.893582 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:44.893476    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="af782ac5-c43b-4ff0-994d-0a2f688ef479" containerName="revoke-keys"
Oct 30 20:23:44.897724 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:44.897701    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659"
Oct 30 20:23:44.903725 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:44.902855    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:44.904674 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:44.904644    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\""
Oct 30 20:23:44.908553 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:44.908532    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:23:44.908797 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:44.908779    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:23:44.911164 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:44.911141    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659"]
Oct 30 20:23:45.077051 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:45.077015    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-226r5\" (UniqueName: \"kubernetes.io/projected/e2640673-8d78-4d5d-ae08-7eb14eae5963-kube-api-access-226r5\") pod \"maas-api-revoke-sub-6c9384b4-vr659\" (UID: \"e2640673-8d78-4d5d-ae08-7eb14eae5963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659"
Oct 30 20:23:45.077267 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:45.077225    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2640673-8d78-4d5d-ae08-7eb14eae5963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c9384b4-vr659\" (UID: \"e2640673-8d78-4d5d-ae08-7eb14eae5963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659"
Oct 30 20:23:45.077379 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:45.077267    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2640673-8d78-4d5d-ae08-7eb14eae5963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c9384b4-vr659\" (UID: \"e2640673-8d78-4d5d-ae08-7eb14eae5963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659"
Oct 30 20:23:45.178667 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:45.178585    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2640673-8d78-4d5d-ae08-7eb14eae5963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c9384b4-vr659\" (UID: \"e2640673-8d78-4d5d-ae08-7eb14eae5963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659"
Oct 30 20:23:45.178667 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:45.178630    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2640673-8d78-4d5d-ae08-7eb14eae5963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c9384b4-vr659\" (UID: \"e2640673-8d78-4d5d-ae08-7eb14eae5963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659"
Oct 30 20:23:45.178916 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:45.178675    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-226r5\" (UniqueName: \"kubernetes.io/projected/e2640673-8d78-4d5d-ae08-7eb14eae5963-kube-api-access-226r5\") pod \"maas-api-revoke-sub-6c9384b4-vr659\" (UID: \"e2640673-8d78-4d5d-ae08-7eb14eae5963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659"
Oct 30 20:23:45.179339 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:45.179311    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2640673-8d78-4d5d-ae08-7eb14eae5963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c9384b4-vr659\" (UID: \"e2640673-8d78-4d5d-ae08-7eb14eae5963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659"
Oct 30 20:23:45.181014 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:45.180984    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2640673-8d78-4d5d-ae08-7eb14eae5963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c9384b4-vr659\" (UID: \"e2640673-8d78-4d5d-ae08-7eb14eae5963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659"
Oct 30 20:23:45.192306 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:45.192287    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-226r5\" (UniqueName: \"kubernetes.io/projected/e2640673-8d78-4d5d-ae08-7eb14eae5963-kube-api-access-226r5\") pod \"maas-api-revoke-sub-6c9384b4-vr659\" (UID: \"e2640673-8d78-4d5d-ae08-7eb14eae5963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659"
Oct 30 20:23:45.215037 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:45.215011    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659"
Oct 30 20:23:45.489505 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:45.489478    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659"]
Oct 30 20:23:45.491137 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:23:45.491107    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2640673_8d78_4d5d_ae08_7eb14eae5963.slice/crio-7d1155303a8ce8f60ba46365e4ee8215aa116109f48d855a21b6545778e0b4e2 WatchSource:0}: Error finding container 7d1155303a8ce8f60ba46365e4ee8215aa116109f48d855a21b6545778e0b4e2: Status 404 returned error can't find the container with id 7d1155303a8ce8f60ba46365e4ee8215aa116109f48d855a21b6545778e0b4e2
Oct 30 20:23:45.882001 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:45.881963    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659" event={"ID":"e2640673-8d78-4d5d-ae08-7eb14eae5963","Type":"ContainerStarted","Data":"8ba2d7e892869eed65dd5054ab7da799eb5430e3c76f4200b5bc8178f4b7fe5a"}
Oct 30 20:23:45.882223 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:45.882036    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659" event={"ID":"e2640673-8d78-4d5d-ae08-7eb14eae5963","Type":"ContainerStarted","Data":"7d1155303a8ce8f60ba46365e4ee8215aa116109f48d855a21b6545778e0b4e2"}
Oct 30 20:23:45.902449 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:45.902402    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659" podStartSLOduration=1.902389975 podStartE2EDuration="1.902389975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:45.901233199 +0000 UTC m=+3104.816840056" watchObservedRunningTime="2026-10-30 20:23:45.902389975 +0000 UTC m=+3104.817996831"
Oct 30 20:23:46.888541 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:46.888501    6030 generic.go:356] "Generic (PLEG): container finished" podID="e2640673-8d78-4d5d-ae08-7eb14eae5963" containerID="8ba2d7e892869eed65dd5054ab7da799eb5430e3c76f4200b5bc8178f4b7fe5a" exitCode=0
Oct 30 20:23:46.888735 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:46.888586    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659" event={"ID":"e2640673-8d78-4d5d-ae08-7eb14eae5963","Type":"ContainerDied","Data":"8ba2d7e892869eed65dd5054ab7da799eb5430e3c76f4200b5bc8178f4b7fe5a"}
Oct 30 20:23:48.024027 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:48.024002    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659"
Oct 30 20:23:48.206136 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:48.206034    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2640673-8d78-4d5d-ae08-7eb14eae5963-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2640673-8d78-4d5d-ae08-7eb14eae5963-maas-api-xks-ca\") pod \"e2640673-8d78-4d5d-ae08-7eb14eae5963\" (UID: \"e2640673-8d78-4d5d-ae08-7eb14eae5963\") "
Oct 30 20:23:48.206136 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:48.206095    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2640673-8d78-4d5d-ae08-7eb14eae5963-kube-api-access-226r5\" (UniqueName: \"kubernetes.io/projected/e2640673-8d78-4d5d-ae08-7eb14eae5963-kube-api-access-226r5\") pod \"e2640673-8d78-4d5d-ae08-7eb14eae5963\" (UID: \"e2640673-8d78-4d5d-ae08-7eb14eae5963\") "
Oct 30 20:23:48.206136 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:48.206123    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2640673-8d78-4d5d-ae08-7eb14eae5963-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2640673-8d78-4d5d-ae08-7eb14eae5963-maas-api-openshift-ca\") pod \"e2640673-8d78-4d5d-ae08-7eb14eae5963\" (UID: \"e2640673-8d78-4d5d-ae08-7eb14eae5963\") "
Oct 30 20:23:48.206546 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:48.206515    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2640673-8d78-4d5d-ae08-7eb14eae5963-maas-api-openshift-ca" pod "e2640673-8d78-4d5d-ae08-7eb14eae5963" (UID: "e2640673-8d78-4d5d-ae08-7eb14eae5963"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:48.208232 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:48.208208    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2640673-8d78-4d5d-ae08-7eb14eae5963-maas-api-xks-ca" pod "e2640673-8d78-4d5d-ae08-7eb14eae5963" (UID: "e2640673-8d78-4d5d-ae08-7eb14eae5963"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:48.208405 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:48.208385    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2640673-8d78-4d5d-ae08-7eb14eae5963-kube-api-access-226r5" pod "e2640673-8d78-4d5d-ae08-7eb14eae5963" (UID: "e2640673-8d78-4d5d-ae08-7eb14eae5963"). InnerVolumeSpecName "kube-api-access-226r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:48.307551 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:48.307519    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2640673-8d78-4d5d-ae08-7eb14eae5963-maas-api-xks-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:48.307729 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:48.307556    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-226r5\" (UniqueName: \"kubernetes.io/projected/e2640673-8d78-4d5d-ae08-7eb14eae5963-kube-api-access-226r5\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:48.307729 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:48.307579    6030 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2640673-8d78-4d5d-ae08-7eb14eae5963-maas-api-openshift-ca\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:48.899743 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:48.899714    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659"
Oct 30 20:23:48.899933 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:48.899745    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9384b4-vr659" event={"ID":"e2640673-8d78-4d5d-ae08-7eb14eae5963","Type":"ContainerDied","Data":"7d1155303a8ce8f60ba46365e4ee8215aa116109f48d855a21b6545778e0b4e2"}
Oct 30 20:23:48.899933 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:48.899797    6030 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d1155303a8ce8f60ba46365e4ee8215aa116109f48d855a21b6545778e0b4e2"
Oct 30 20:23:52.680580 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:52.680540    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"]
Oct 30 20:23:52.680981 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:52.680756    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr" podUID="a57ba962-389a-4798-a788-cb0dedfa0f5c" containerName="istio-proxy" containerID="cri-o://6c53871af6c606bd4fec21f6becc9eebb62a76cb102ff366a0f53a6573b85af8" gracePeriod=30
Oct 30 20:23:53.797131 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:53.797038    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" containerID="cri-o://76ef8f53ed0b46e2612ad4d80f5aac87888b61e099f13bf6c18baa3568973bd7" gracePeriod=45
Oct 30 20:23:53.809503 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:53.809448    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main" containerID="cri-o://9645fdfaa19212d346da07306e1aba1e5bdaa61c888d57880b9601877d3aa98b" gracePeriod=45
Oct 30 20:23:53.825006 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:53.824975    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" containerID="cri-o://348340ebe6ce66dfc18a6abef92c645d950010ff86e1cbdfbc5030401b915906" gracePeriod=45
Oct 30 20:23:53.831683 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:53.831653    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": dial tcp 10.133.0.37:8000: connect: connection refused"
Oct 30 20:23:53.924597 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:53.924562    6030 generic.go:356] "Generic (PLEG): container finished" podID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerID="76ef8f53ed0b46e2612ad4d80f5aac87888b61e099f13bf6c18baa3568973bd7" exitCode=0
Oct 30 20:23:53.924737 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:53.924632    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" event={"ID":"a3be41ac-1e27-4393-a083-c8a64286d53f","Type":"ContainerDied","Data":"76ef8f53ed0b46e2612ad4d80f5aac87888b61e099f13bf6c18baa3568973bd7"}
Oct 30 20:23:53.926177 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:53.926158    6030 generic.go:356] "Generic (PLEG): container finished" podID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerID="9645fdfaa19212d346da07306e1aba1e5bdaa61c888d57880b9601877d3aa98b" exitCode=0
Oct 30 20:23:53.926268 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:53.926191    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" event={"ID":"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9","Type":"ContainerDied","Data":"9645fdfaa19212d346da07306e1aba1e5bdaa61c888d57880b9601877d3aa98b"}
Oct 30 20:23:54.082109 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.082036    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 20:23:54.085156 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.085115    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 20:23:54.110014 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.109982    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 20:23:54.156635 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.156598    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-model-cache\") pod \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") "
Oct 30 20:23:54.156822 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.156654    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-kube-api-access-rdc2s\" (UniqueName: \"kubernetes.io/projected/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-kube-api-access-rdc2s\") pod \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") "
Oct 30 20:23:54.156822 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.156687    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-tmp-dir\") pod \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") "
Oct 30 20:23:54.156822 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.156714    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-dshm\") pod \"a3be41ac-1e27-4393-a083-c8a64286d53f\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") "
Oct 30 20:23:54.156822 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.156739    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-tls-certs\" (UniqueName: \"kubernetes.io/secret/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-tls-certs\") pod \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") "
Oct 30 20:23:54.156822 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.156785    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-kube-api-access-8j8d6\" (UniqueName: \"kubernetes.io/projected/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-kube-api-access-8j8d6\") pod \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") "
Oct 30 20:23:54.156822 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.156812    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3be41ac-1e27-4393-a083-c8a64286d53f-kube-api-access-7b5gn\" (UniqueName: \"kubernetes.io/projected/a3be41ac-1e27-4393-a083-c8a64286d53f-kube-api-access-7b5gn\") pod \"a3be41ac-1e27-4393-a083-c8a64286d53f\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") "
Oct 30 20:23:54.157304 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.156842    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-tmp-dir\") pod \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") "
Oct 30 20:23:54.157304 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.156883    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-dshm\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-dshm\") pod \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") "
Oct 30 20:23:54.157304 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.156914    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-home\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-home\") pod \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") "
Oct 30 20:23:54.157304 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.156968    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-home\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-home\") pod \"a3be41ac-1e27-4393-a083-c8a64286d53f\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") "
Oct 30 20:23:54.157304 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.157000    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-model-cache\") pod \"a3be41ac-1e27-4393-a083-c8a64286d53f\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") "
Oct 30 20:23:54.157304 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.157031    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-home\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-home\") pod \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\" (UID: \"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545\") "
Oct 30 20:23:54.157304 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.157099    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3be41ac-1e27-4393-a083-c8a64286d53f-tls-certs\" (UniqueName: \"kubernetes.io/secret/a3be41ac-1e27-4393-a083-c8a64286d53f-tls-certs\") pod \"a3be41ac-1e27-4393-a083-c8a64286d53f\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") "
Oct 30 20:23:54.157304 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.157131    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-dshm\") pod \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") "
Oct 30 20:23:54.157304 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.157201    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-tls-certs\" (UniqueName: \"kubernetes.io/secret/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-tls-certs\") pod \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") "
Oct 30 20:23:54.157304 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.157228    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-model-cache\") pod \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\" (UID: \"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9\") "
Oct 30 20:23:54.157304 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.157280    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-tmp-dir\") pod \"a3be41ac-1e27-4393-a083-c8a64286d53f\" (UID: \"a3be41ac-1e27-4393-a083-c8a64286d53f\") "
Oct 30 20:23:54.157818 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.157792    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-tmp-dir" pod "a3be41ac-1e27-4393-a083-c8a64286d53f" (UID: "a3be41ac-1e27-4393-a083-c8a64286d53f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:54.158014 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.157986    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-tmp-dir" pod "f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" (UID: "f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:54.158527 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.158502    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-home" pod "19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" (UID: "19a0c0d2-60d9-4d3c-b88b-bdd5a661f545"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:54.158756 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.158730    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-home" pod "f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" (UID: "f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:54.158970 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.158948    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-home" pod "a3be41ac-1e27-4393-a083-c8a64286d53f" (UID: "a3be41ac-1e27-4393-a083-c8a64286d53f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:54.159210 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.159184    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-model-cache" pod "a3be41ac-1e27-4393-a083-c8a64286d53f" (UID: "a3be41ac-1e27-4393-a083-c8a64286d53f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:54.159384 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.159217    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-model-cache" pod "19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" (UID: "19a0c0d2-60d9-4d3c-b88b-bdd5a661f545"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:54.159513 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.159382    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-model-cache" pod "f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" (UID: "f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:54.163111 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.162942    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3be41ac-1e27-4393-a083-c8a64286d53f-kube-api-access-7b5gn" pod "a3be41ac-1e27-4393-a083-c8a64286d53f" (UID: "a3be41ac-1e27-4393-a083-c8a64286d53f"). InnerVolumeSpecName "kube-api-access-7b5gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:54.163207 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.163133    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-tmp-dir" pod "19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" (UID: "19a0c0d2-60d9-4d3c-b88b-bdd5a661f545"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:54.163467 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.163434    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-dshm" pod "19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" (UID: "19a0c0d2-60d9-4d3c-b88b-bdd5a661f545"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:54.164258 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.164232    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-tls-certs" pod "19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" (UID: "19a0c0d2-60d9-4d3c-b88b-bdd5a661f545"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:54.165258 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.165115    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-kube-api-access-rdc2s" pod "19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" (UID: "19a0c0d2-60d9-4d3c-b88b-bdd5a661f545"). InnerVolumeSpecName "kube-api-access-rdc2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:54.165258 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.165224    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3be41ac-1e27-4393-a083-c8a64286d53f-tls-certs" pod "a3be41ac-1e27-4393-a083-c8a64286d53f" (UID: "a3be41ac-1e27-4393-a083-c8a64286d53f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:54.165258 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.165235    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-dshm" pod "f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" (UID: "f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:54.166755 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.166730    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-tls-certs" pod "f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" (UID: "f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:54.167012 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.166987    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-dshm" pod "a3be41ac-1e27-4393-a083-c8a64286d53f" (UID: "a3be41ac-1e27-4393-a083-c8a64286d53f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:54.167441 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.167413    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-kube-api-access-8j8d6" pod "f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" (UID: "f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9"). InnerVolumeSpecName "kube-api-access-8j8d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:54.258105 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258068    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.258105 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258104    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.258301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258117    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.258301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258126    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.258301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258135    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdc2s\" (UniqueName: \"kubernetes.io/projected/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-kube-api-access-rdc2s\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.258301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258145    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.258301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258153    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.258301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258160    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.258301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258168    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j8d6\" (UniqueName: \"kubernetes.io/projected/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-kube-api-access-8j8d6\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.258301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258177    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b5gn\" (UniqueName: \"kubernetes.io/projected/a3be41ac-1e27-4393-a083-c8a64286d53f-kube-api-access-7b5gn\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.258301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258185    6030 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-tmp-dir\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.258301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258193    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.258301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258201    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.258301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258208    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.258301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258215    6030 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3be41ac-1e27-4393-a083-c8a64286d53f-model-cache\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.258301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258224    6030 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545-home\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.258301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258232    6030 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3be41ac-1e27-4393-a083-c8a64286d53f-tls-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.258301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.258239    6030 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9-dshm\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.932260 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.932226    6030 generic.go:356] "Generic (PLEG): container finished" podID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerID="348340ebe6ce66dfc18a6abef92c645d950010ff86e1cbdfbc5030401b915906" exitCode=0
Oct 30 20:23:54.932726 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.932271    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"
Oct 30 20:23:54.932726 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.932308    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" event={"ID":"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545","Type":"ContainerDied","Data":"348340ebe6ce66dfc18a6abef92c645d950010ff86e1cbdfbc5030401b915906"}
Oct 30 20:23:54.932726 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.932365    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv" event={"ID":"19a0c0d2-60d9-4d3c-b88b-bdd5a661f545","Type":"ContainerDied","Data":"f2f2985343fb3cb215349e41bd62e6d564a18d813a89b6e7350f5ba0109b4777"}
Oct 30 20:23:54.932726 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.932391    6030 scope.go:122] "RemoveContainer" containerID="348340ebe6ce66dfc18a6abef92c645d950010ff86e1cbdfbc5030401b915906"
Oct 30 20:23:54.933868 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.933846    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" event={"ID":"a3be41ac-1e27-4393-a083-c8a64286d53f","Type":"ContainerDied","Data":"a27efb62dd72d22d4f2fab80b5dbace0c9a63e4835646a1b916eb21d5570e643"}
Oct 30 20:23:54.933941 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.933873    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"
Oct 30 20:23:54.935375 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.935355    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5" event={"ID":"f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9","Type":"ContainerDied","Data":"922f1879fbe19d19d75e5e9bce437316f4b7a81b71cc6cfa3808e8d48005d925"}
Oct 30 20:23:54.935493 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.935385    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"
Oct 30 20:23:54.941626 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.941607    6030 scope.go:122] "RemoveContainer" containerID="348340ebe6ce66dfc18a6abef92c645d950010ff86e1cbdfbc5030401b915906"
Oct 30 20:23:54.941940 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:23:54.941914    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348340ebe6ce66dfc18a6abef92c645d950010ff86e1cbdfbc5030401b915906\": container with ID starting with 348340ebe6ce66dfc18a6abef92c645d950010ff86e1cbdfbc5030401b915906 not found: ID does not exist" containerID="348340ebe6ce66dfc18a6abef92c645d950010ff86e1cbdfbc5030401b915906"
Oct 30 20:23:54.942080 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.941951    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348340ebe6ce66dfc18a6abef92c645d950010ff86e1cbdfbc5030401b915906"} err="failed to get container status \"348340ebe6ce66dfc18a6abef92c645d950010ff86e1cbdfbc5030401b915906\": rpc error: code = NotFound desc = could not find container \"348340ebe6ce66dfc18a6abef92c645d950010ff86e1cbdfbc5030401b915906\": container with ID starting with 348340ebe6ce66dfc18a6abef92c645d950010ff86e1cbdfbc5030401b915906 not found: ID does not exist"
Oct 30 20:23:54.942080 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.941973    6030 scope.go:122] "RemoveContainer" containerID="76ef8f53ed0b46e2612ad4d80f5aac87888b61e099f13bf6c18baa3568973bd7"
Oct 30 20:23:54.951448 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.951431    6030 scope.go:122] "RemoveContainer" containerID="9645fdfaa19212d346da07306e1aba1e5bdaa61c888d57880b9601877d3aa98b"
Oct 30 20:23:54.962916 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.962896    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"]
Oct 30 20:23:54.967395 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.967377    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk"]
Oct 30 20:23:54.981253 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.981234    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"]
Oct 30 20:23:54.986495 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.986477    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/premium-w0-8836a8-kserve-68bbcf8f58-nxpg5"]
Oct 30 20:23:55.000003 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:54.999982    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"]
Oct 30 20:23:55.015876 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:55.015853    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/simulator-w0-8836a8-kserve-689b4899f-p4xcv"]
Oct 30 20:23:55.071233 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:55.071175    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-w0-8836a8-kserve-7c66db8877-zldpk" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 20:23:55.199199 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:55.199169    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" path="/var/lib/kubelet/pods/19a0c0d2-60d9-4d3c-b88b-bdd5a661f545/volumes"
Oct 30 20:23:55.199543 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:55.199531    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" path="/var/lib/kubelet/pods/a3be41ac-1e27-4393-a083-c8a64286d53f/volumes"
Oct 30 20:23:55.199881 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:55.199870    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" path="/var/lib/kubelet/pods/f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9/volumes"
Oct 30 20:23:57.926396 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.926370    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:23:57.962174 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.962034    6030 generic.go:356] "Generic (PLEG): container finished" podID="a57ba962-389a-4798-a788-cb0dedfa0f5c" containerID="6c53871af6c606bd4fec21f6becc9eebb62a76cb102ff366a0f53a6573b85af8" exitCode=0
Oct 30 20:23:57.962174 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.962092    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr" event={"ID":"a57ba962-389a-4798-a788-cb0dedfa0f5c","Type":"ContainerDied","Data":"6c53871af6c606bd4fec21f6becc9eebb62a76cb102ff366a0f53a6573b85af8"}
Oct 30 20:23:57.962174 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.962116    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"
Oct 30 20:23:57.962174 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.962157    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr" event={"ID":"a57ba962-389a-4798-a788-cb0dedfa0f5c","Type":"ContainerDied","Data":"c3e37570b11c4d903de1e9e10fd902e5805d2f4a23503cee95a3112a9b7173cd"}
Oct 30 20:23:57.962503 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.962191    6030 scope.go:122] "RemoveContainer" containerID="6c53871af6c606bd4fec21f6becc9eebb62a76cb102ff366a0f53a6573b85af8"
Oct 30 20:23:57.971700 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.971673    6030 scope.go:122] "RemoveContainer" containerID="6c53871af6c606bd4fec21f6becc9eebb62a76cb102ff366a0f53a6573b85af8"
Oct 30 20:23:57.971953 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:23:57.971933    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c53871af6c606bd4fec21f6becc9eebb62a76cb102ff366a0f53a6573b85af8\": container with ID starting with 6c53871af6c606bd4fec21f6becc9eebb62a76cb102ff366a0f53a6573b85af8 not found: ID does not exist" containerID="6c53871af6c606bd4fec21f6becc9eebb62a76cb102ff366a0f53a6573b85af8"
Oct 30 20:23:57.972264 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.971960    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c53871af6c606bd4fec21f6becc9eebb62a76cb102ff366a0f53a6573b85af8"} err="failed to get container status \"6c53871af6c606bd4fec21f6becc9eebb62a76cb102ff366a0f53a6573b85af8\": rpc error: code = NotFound desc = could not find container \"6c53871af6c606bd4fec21f6becc9eebb62a76cb102ff366a0f53a6573b85af8\": container with ID starting with 6c53871af6c606bd4fec21f6becc9eebb62a76cb102ff366a0f53a6573b85af8 not found: ID does not exist"
Oct 30 20:23:57.989188 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.989165    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-workload-certs\") pod \"a57ba962-389a-4798-a788-cb0dedfa0f5c\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") "
Oct 30 20:23:57.989288 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.989204    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a57ba962-389a-4798-a788-cb0dedfa0f5c-kube-api-access-cbfzj\" (UniqueName: \"kubernetes.io/projected/a57ba962-389a-4798-a788-cb0dedfa0f5c-kube-api-access-cbfzj\") pod \"a57ba962-389a-4798-a788-cb0dedfa0f5c\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") "
Oct 30 20:23:57.989288 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.989240    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-podinfo\") pod \"a57ba962-389a-4798-a788-cb0dedfa0f5c\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") "
Oct 30 20:23:57.989288 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.989260    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-data\") pod \"a57ba962-389a-4798-a788-cb0dedfa0f5c\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") "
Oct 30 20:23:57.989447 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.989289    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-envoy\") pod \"a57ba962-389a-4798-a788-cb0dedfa0f5c\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") "
Oct 30 20:23:57.989447 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.989323    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-workload-socket\") pod \"a57ba962-389a-4798-a788-cb0dedfa0f5c\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") "
Oct 30 20:23:57.989447 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.989366    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-token\" (UniqueName: \"kubernetes.io/projected/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-token\") pod \"a57ba962-389a-4798-a788-cb0dedfa0f5c\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") "
Oct 30 20:23:57.989447 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.989427    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-credential-socket\") pod \"a57ba962-389a-4798-a788-cb0dedfa0f5c\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") "
Oct 30 20:23:57.989672 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.989438    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-workload-certs" pod "a57ba962-389a-4798-a788-cb0dedfa0f5c" (UID: "a57ba962-389a-4798-a788-cb0dedfa0f5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:57.989672 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.989459    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a57ba962-389a-4798-a788-cb0dedfa0f5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a57ba962-389a-4798-a788-cb0dedfa0f5c-istiod-ca-cert\") pod \"a57ba962-389a-4798-a788-cb0dedfa0f5c\" (UID: \"a57ba962-389a-4798-a788-cb0dedfa0f5c\") "
Oct 30 20:23:57.989672 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.989596    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-data" pod "a57ba962-389a-4798-a788-cb0dedfa0f5c" (UID: "a57ba962-389a-4798-a788-cb0dedfa0f5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:57.989672 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.989611    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:57.989672 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.989626    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-workload-socket" pod "a57ba962-389a-4798-a788-cb0dedfa0f5c" (UID: "a57ba962-389a-4798-a788-cb0dedfa0f5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:57.989918 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.989822    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-credential-socket" pod "a57ba962-389a-4798-a788-cb0dedfa0f5c" (UID: "a57ba962-389a-4798-a788-cb0dedfa0f5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:57.989961 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.989947    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a57ba962-389a-4798-a788-cb0dedfa0f5c-istiod-ca-cert" pod "a57ba962-389a-4798-a788-cb0dedfa0f5c" (UID: "a57ba962-389a-4798-a788-cb0dedfa0f5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:57.991633 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.991609    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57ba962-389a-4798-a788-cb0dedfa0f5c-kube-api-access-cbfzj" pod "a57ba962-389a-4798-a788-cb0dedfa0f5c" (UID: "a57ba962-389a-4798-a788-cb0dedfa0f5c"). InnerVolumeSpecName "kube-api-access-cbfzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:57.991787 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.991772    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-envoy" pod "a57ba962-389a-4798-a788-cb0dedfa0f5c" (UID: "a57ba962-389a-4798-a788-cb0dedfa0f5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:57.991930 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.991909    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-token" pod "a57ba962-389a-4798-a788-cb0dedfa0f5c" (UID: "a57ba962-389a-4798-a788-cb0dedfa0f5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:57.992027 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:57.992013    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-podinfo" pod "a57ba962-389a-4798-a788-cb0dedfa0f5c" (UID: "a57ba962-389a-4798-a788-cb0dedfa0f5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:58.091135 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:58.091092    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:58.091135 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:58.091129    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:58.091135 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:58.091140    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:58.091463 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:58.091150    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a57ba962-389a-4798-a788-cb0dedfa0f5c-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:58.091463 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:58.091160    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbfzj\" (UniqueName: \"kubernetes.io/projected/a57ba962-389a-4798-a788-cb0dedfa0f5c-kube-api-access-cbfzj\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:58.091463 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:58.091168    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:58.091463 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:58.091177    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:58.091463 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:58.091186    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a57ba962-389a-4798-a788-cb0dedfa0f5c-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:58.287302 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:58.287267    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"]
Oct 30 20:23:58.292034 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:58.292001    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mhqnr"]
Oct 30 20:23:59.198751 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:23:59.198721    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a57ba962-389a-4798-a788-cb0dedfa0f5c" path="/var/lib/kubelet/pods/a57ba962-389a-4798-a788-cb0dedfa0f5c/volumes"
Oct 30 20:24:00.216304 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:00.216259    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"]
Oct 30 20:24:00.219114 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:00.219086    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8836a8-wfg4n"]
Oct 30 20:24:01.201899 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:01.201861    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af782ac5-c43b-4ff0-994d-0a2f688ef479" path="/var/lib/kubelet/pods/af782ac5-c43b-4ff0-994d-0a2f688ef479/volumes"
Oct 30 20:24:01.406886 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:01.406851    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"]
Oct 30 20:24:01.407324 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:01.407076    6030 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh" podUID="becea00a-6860-4f4e-830f-6f504d0cf8b8" containerName="istio-proxy" containerID="cri-o://5fa8209ac1607463dc5f87b61c38b35342d0fb77ae0124f27cbb80f835799ca1" gracePeriod=30
Oct 30 20:24:04.285511 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:24:04.285467    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:24:05.869684 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:05.869646    6030 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:24:05.870143 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:05.869700    6030 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh" podUID="becea00a-6860-4f4e-830f-6f504d0cf8b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:24:06.647615 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.647591    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 20:24:06.763201 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763113    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-data\") pod \"becea00a-6860-4f4e-830f-6f504d0cf8b8\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") "
Oct 30 20:24:06.763201 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763170    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-workload-socket\") pod \"becea00a-6860-4f4e-830f-6f504d0cf8b8\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") "
Oct 30 20:24:06.763201 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763199    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-credential-socket\") pod \"becea00a-6860-4f4e-830f-6f504d0cf8b8\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") "
Oct 30 20:24:06.763490 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763254    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/becea00a-6860-4f4e-830f-6f504d0cf8b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/becea00a-6860-4f4e-830f-6f504d0cf8b8-istiod-ca-cert\") pod \"becea00a-6860-4f4e-830f-6f504d0cf8b8\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") "
Oct 30 20:24:06.763490 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763289    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-workload-certs\") pod \"becea00a-6860-4f4e-830f-6f504d0cf8b8\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") "
Oct 30 20:24:06.763490 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763320    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-podinfo\") pod \"becea00a-6860-4f4e-830f-6f504d0cf8b8\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") "
Oct 30 20:24:06.763490 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763350    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/becea00a-6860-4f4e-830f-6f504d0cf8b8-kube-api-access-pjtsw\" (UniqueName: \"kubernetes.io/projected/becea00a-6860-4f4e-830f-6f504d0cf8b8-kube-api-access-pjtsw\") pod \"becea00a-6860-4f4e-830f-6f504d0cf8b8\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") "
Oct 30 20:24:06.763490 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763394    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-token\" (UniqueName: \"kubernetes.io/projected/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-token\") pod \"becea00a-6860-4f4e-830f-6f504d0cf8b8\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") "
Oct 30 20:24:06.763490 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763440    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-workload-socket" pod "becea00a-6860-4f4e-830f-6f504d0cf8b8" (UID: "becea00a-6860-4f4e-830f-6f504d0cf8b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:06.763490 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763450    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-data" pod "becea00a-6860-4f4e-830f-6f504d0cf8b8" (UID: "becea00a-6860-4f4e-830f-6f504d0cf8b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:06.763490 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763463    6030 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-envoy\") pod \"becea00a-6860-4f4e-830f-6f504d0cf8b8\" (UID: \"becea00a-6860-4f4e-830f-6f504d0cf8b8\") "
Oct 30 20:24:06.763829 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763610    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-credential-socket" pod "becea00a-6860-4f4e-830f-6f504d0cf8b8" (UID: "becea00a-6860-4f4e-830f-6f504d0cf8b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:06.763829 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763656    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-workload-certs" pod "becea00a-6860-4f4e-830f-6f504d0cf8b8" (UID: "becea00a-6860-4f4e-830f-6f504d0cf8b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:06.763829 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763682    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/becea00a-6860-4f4e-830f-6f504d0cf8b8-istiod-ca-cert" pod "becea00a-6860-4f4e-830f-6f504d0cf8b8" (UID: "becea00a-6860-4f4e-830f-6f504d0cf8b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:06.763829 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763747    6030 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-data\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:06.763829 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763766    6030 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-workload-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:06.763829 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763781    6030 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-credential-socket\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:06.763829 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763795    6030 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/becea00a-6860-4f4e-830f-6f504d0cf8b8-istiod-ca-cert\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:06.763829 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.763808    6030 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-workload-certs\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:06.765878 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.765850    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-token" pod "becea00a-6860-4f4e-830f-6f504d0cf8b8" (UID: "becea00a-6860-4f4e-830f-6f504d0cf8b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:06.766014 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.765880    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/becea00a-6860-4f4e-830f-6f504d0cf8b8-kube-api-access-pjtsw" pod "becea00a-6860-4f4e-830f-6f504d0cf8b8" (UID: "becea00a-6860-4f4e-830f-6f504d0cf8b8"). InnerVolumeSpecName "kube-api-access-pjtsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:06.766133 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.766110    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-envoy" pod "becea00a-6860-4f4e-830f-6f504d0cf8b8" (UID: "becea00a-6860-4f4e-830f-6f504d0cf8b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:06.766203 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.766110    6030 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-podinfo" pod "becea00a-6860-4f4e-830f-6f504d0cf8b8" (UID: "becea00a-6860-4f4e-830f-6f504d0cf8b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:24:06.864816 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.864785    6030 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-podinfo\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:06.864816 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.864813    6030 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjtsw\" (UniqueName: \"kubernetes.io/projected/becea00a-6860-4f4e-830f-6f504d0cf8b8-kube-api-access-pjtsw\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:06.864993 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.864823    6030 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-token\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:06.864993 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:06.864832    6030 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/becea00a-6860-4f4e-830f-6f504d0cf8b8-istio-envoy\") on node \"ip-10-0-142-167.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:07.007590 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:07.007556    6030 generic.go:356] "Generic (PLEG): container finished" podID="becea00a-6860-4f4e-830f-6f504d0cf8b8" containerID="5fa8209ac1607463dc5f87b61c38b35342d0fb77ae0124f27cbb80f835799ca1" exitCode=0
Oct 30 20:24:07.007998 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:07.007608    6030 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"
Oct 30 20:24:07.007998 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:07.007610    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh" event={"ID":"becea00a-6860-4f4e-830f-6f504d0cf8b8","Type":"ContainerDied","Data":"5fa8209ac1607463dc5f87b61c38b35342d0fb77ae0124f27cbb80f835799ca1"}
Oct 30 20:24:07.007998 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:07.007732    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh" event={"ID":"becea00a-6860-4f4e-830f-6f504d0cf8b8","Type":"ContainerDied","Data":"7a8b4d5a24acb62c1a27f3d835098dae1411b86a707b9b16bc4de89ae425a6da"}
Oct 30 20:24:07.007998 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:07.007760    6030 scope.go:122] "RemoveContainer" containerID="5fa8209ac1607463dc5f87b61c38b35342d0fb77ae0124f27cbb80f835799ca1"
Oct 30 20:24:07.016859 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:07.016844    6030 scope.go:122] "RemoveContainer" containerID="5fa8209ac1607463dc5f87b61c38b35342d0fb77ae0124f27cbb80f835799ca1"
Oct 30 20:24:07.017222 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:24:07.017202    6030 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa8209ac1607463dc5f87b61c38b35342d0fb77ae0124f27cbb80f835799ca1\": container with ID starting with 5fa8209ac1607463dc5f87b61c38b35342d0fb77ae0124f27cbb80f835799ca1 not found: ID does not exist" containerID="5fa8209ac1607463dc5f87b61c38b35342d0fb77ae0124f27cbb80f835799ca1"
Oct 30 20:24:07.017293 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:07.017233    6030 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa8209ac1607463dc5f87b61c38b35342d0fb77ae0124f27cbb80f835799ca1"} err="failed to get container status \"5fa8209ac1607463dc5f87b61c38b35342d0fb77ae0124f27cbb80f835799ca1\": rpc error: code = NotFound desc = could not find container \"5fa8209ac1607463dc5f87b61c38b35342d0fb77ae0124f27cbb80f835799ca1\": container with ID starting with 5fa8209ac1607463dc5f87b61c38b35342d0fb77ae0124f27cbb80f835799ca1 not found: ID does not exist"
Oct 30 20:24:07.031816 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:07.031786    6030 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"]
Oct 30 20:24:07.036648 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:07.036626    6030 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-4xjgh"]
Oct 30 20:24:07.199773 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:07.199741    6030 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="becea00a-6860-4f4e-830f-6f504d0cf8b8" path="/var/lib/kubelet/pods/becea00a-6860-4f4e-830f-6f504d0cf8b8/volumes"
Oct 30 20:24:09.314526 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:09.314496    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mgclc_7d40f8de-854b-4a46-bf24-a37b49e75583/manager/0.log"
Oct 30 20:24:10.520733 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:10.520702    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6pvgl_7f8b5c02-97b9-4018-ab0a-43af4476f135/manager/0.log"
Oct 30 20:24:11.527798 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:11.527763    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-grxfc_39be0ba6-63fe-4329-bfbd-7eed3d36be55/manager/0.log"
Oct 30 20:24:11.954774 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:11.954747    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l7rcc_2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc/manager/0.log"
Oct 30 20:24:12.430061 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:12.430018    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cf964748-d8zhr_c2b7322b-ec51-4436-bedb-b4a923ffdf75/manager/1.log"
Oct 30 20:24:13.024484 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:13.024453    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-89xm5_6a9b909a-d885-4ff2-8032-8079ac535f5a/proxy/0.log"
Oct 30 20:24:13.417135 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:13.417105    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cf964748-d8zhr_c2b7322b-ec51-4436-bedb-b4a923ffdf75/manager/0.log"
Oct 30 20:24:24.899887 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:24.899855    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-grxfc_39be0ba6-63fe-4329-bfbd-7eed3d36be55/manager/0.log"
Oct 30 20:24:25.328277 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:25.328184    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l7rcc_2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc/manager/0.log"
Oct 30 20:24:25.795576 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:25.795540    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cf964748-d8zhr_c2b7322b-ec51-4436-bedb-b4a923ffdf75/manager/1.log"
Oct 30 20:24:26.372033 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:26.372001    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-89xm5_6a9b909a-d885-4ff2-8032-8079ac535f5a/proxy/0.log"
Oct 30 20:24:26.786980 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:26.786951    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cf964748-d8zhr_c2b7322b-ec51-4436-bedb-b4a923ffdf75/manager/0.log"
Oct 30 20:24:27.559674 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:27.559643    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0762643f-zlvtb_675d9d43-b63c-4eba-8792-034e78ab738e/revoke-keys/0.log"
Oct 30 20:24:27.916572 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:27.916540    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6c9384b4-vr659_e2640673-8d78-4d5d-ae08-7eb14eae5963/revoke-keys/0.log"
Oct 30 20:24:30.939991 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:30.939963    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mgclc_7d40f8de-854b-4a46-bf24-a37b49e75583/manager/0.log"
Oct 30 20:24:32.153087 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:32.153039    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6pvgl_7f8b5c02-97b9-4018-ab0a-43af4476f135/manager/0.log"
Oct 30 20:24:33.443119 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:33.443077    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b99845f6f-rqs6w_2896aa38-c98c-4fa1-a8dd-3149d45e93b6/kube-auth-proxy/0.log"
Oct 30 20:24:34.000024 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:33.999952    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-clbw6_948623f8-d637-44f2-a460-43737e3384d1/istio-proxy/0.log"
Oct 30 20:24:34.287595 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:24:34.287485    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:24:34.383881 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:34.383855    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-x6hbg_db939b9a-f127-428e-bc8f-c137346b089c/istio-proxy/0.log"
Oct 30 20:24:47.250453 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250359    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rkv8j/must-gather-jxd97"]
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250708    6030 cpu_manager.go:414] "containerMap: removing container" podUID="e2640673-8d78-4d5d-ae08-7eb14eae5963" containerName="revoke-keys"
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250727    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2640673-8d78-4d5d-ae08-7eb14eae5963" containerName="revoke-keys"
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250742    6030 cpu_manager.go:414] "containerMap: removing container" podUID="becea00a-6860-4f4e-830f-6f504d0cf8b8" containerName="istio-proxy"
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250747    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="becea00a-6860-4f4e-830f-6f504d0cf8b8" containerName="istio-proxy"
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250757    6030 cpu_manager.go:414] "containerMap: removing container" podUID="a57ba962-389a-4798-a788-cb0dedfa0f5c" containerName="istio-proxy"
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250762    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a57ba962-389a-4798-a788-cb0dedfa0f5c" containerName="istio-proxy"
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250779    6030 cpu_manager.go:414] "containerMap: removing container" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main"
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250783    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main"
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250790    6030 cpu_manager.go:414] "containerMap: removing container" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main"
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250794    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main"
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250808    6030 cpu_manager.go:414] "containerMap: removing container" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main"
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250812    6030 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main"
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250864    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3be41ac-1e27-4393-a083-c8a64286d53f" containerName="main"
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250874    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2640673-8d78-4d5d-ae08-7eb14eae5963" containerName="revoke-keys"
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250880    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="becea00a-6860-4f4e-830f-6f504d0cf8b8" containerName="istio-proxy"
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250885    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="a57ba962-389a-4798-a788-cb0dedfa0f5c" containerName="istio-proxy"
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250893    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a0c0d2-60d9-4d3c-b88b-bdd5a661f545" containerName="main"
Oct 30 20:24:47.250943 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.250898    6030 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8063f3b-1086-4b3e-9364-b0e9a5f8bfc9" containerName="main"
Oct 30 20:24:47.253632 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.253616    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rkv8j/must-gather-jxd97"
Oct 30 20:24:47.258908 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.258869    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rkv8j\"/\"openshift-service-ca.crt\""
Oct 30 20:24:47.259082 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.258906    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rkv8j\"/\"kube-root-ca.crt\""
Oct 30 20:24:47.264762 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.264745    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rkv8j\"/\"default-dockercfg-mwlz6\""
Oct 30 20:24:47.271670 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.271646    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rkv8j/must-gather-jxd97"]
Oct 30 20:24:47.300787 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.300760    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjv56\" (UniqueName: \"kubernetes.io/projected/b49e9a80-6655-44d3-bf4e-183a304589bd-kube-api-access-vjv56\") pod \"must-gather-jxd97\" (UID: \"b49e9a80-6655-44d3-bf4e-183a304589bd\") " pod="openshift-must-gather-rkv8j/must-gather-jxd97"
Oct 30 20:24:47.300890 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.300811    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b49e9a80-6655-44d3-bf4e-183a304589bd-must-gather-output\") pod \"must-gather-jxd97\" (UID: \"b49e9a80-6655-44d3-bf4e-183a304589bd\") " pod="openshift-must-gather-rkv8j/must-gather-jxd97"
Oct 30 20:24:47.358247 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.358220    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4jlph/must-gather-wnx25"]
Oct 30 20:24:47.361613 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.361597    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jlph/must-gather-wnx25"
Oct 30 20:24:47.372698 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.372673    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4jlph\"/\"openshift-service-ca.crt\""
Oct 30 20:24:47.372864 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.372840    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4jlph\"/\"kube-root-ca.crt\""
Oct 30 20:24:47.374142 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.374117    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4jlph\"/\"default-dockercfg-q7rn9\""
Oct 30 20:24:47.375386 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.375365    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5qnpg/must-gather-w769m"]
Oct 30 20:24:47.379149 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.379131    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jlph/must-gather-wnx25"]
Oct 30 20:24:47.379239 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.379222    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5qnpg/must-gather-w769m"
Oct 30 20:24:47.387353 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.387334    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5qnpg/must-gather-w769m"]
Oct 30 20:24:47.401638 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.401612    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b49e9a80-6655-44d3-bf4e-183a304589bd-must-gather-output\") pod \"must-gather-jxd97\" (UID: \"b49e9a80-6655-44d3-bf4e-183a304589bd\") " pod="openshift-must-gather-rkv8j/must-gather-jxd97"
Oct 30 20:24:47.401770 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.401656    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d343c28e-bc12-4ad4-858f-516811d01ce4-must-gather-output\") pod \"must-gather-w769m\" (UID: \"d343c28e-bc12-4ad4-858f-516811d01ce4\") " pod="openshift-must-gather-5qnpg/must-gather-w769m"
Oct 30 20:24:47.401770 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.401724    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rbzn\" (UniqueName: \"kubernetes.io/projected/d343c28e-bc12-4ad4-858f-516811d01ce4-kube-api-access-7rbzn\") pod \"must-gather-w769m\" (UID: \"d343c28e-bc12-4ad4-858f-516811d01ce4\") " pod="openshift-must-gather-5qnpg/must-gather-w769m"
Oct 30 20:24:47.401770 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.401763    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjv56\" (UniqueName: \"kubernetes.io/projected/b49e9a80-6655-44d3-bf4e-183a304589bd-kube-api-access-vjv56\") pod \"must-gather-jxd97\" (UID: \"b49e9a80-6655-44d3-bf4e-183a304589bd\") " pod="openshift-must-gather-rkv8j/must-gather-jxd97"
Oct 30 20:24:47.401905 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.401790    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t667\" (UniqueName: \"kubernetes.io/projected/735ce979-15c1-43fc-a42a-9d5bf7504446-kube-api-access-9t667\") pod \"must-gather-wnx25\" (UID: \"735ce979-15c1-43fc-a42a-9d5bf7504446\") " pod="openshift-must-gather-4jlph/must-gather-wnx25"
Oct 30 20:24:47.401905 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.401828    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/735ce979-15c1-43fc-a42a-9d5bf7504446-must-gather-output\") pod \"must-gather-wnx25\" (UID: \"735ce979-15c1-43fc-a42a-9d5bf7504446\") " pod="openshift-must-gather-4jlph/must-gather-wnx25"
Oct 30 20:24:47.401971 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.401949    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b49e9a80-6655-44d3-bf4e-183a304589bd-must-gather-output\") pod \"must-gather-jxd97\" (UID: \"b49e9a80-6655-44d3-bf4e-183a304589bd\") " pod="openshift-must-gather-rkv8j/must-gather-jxd97"
Oct 30 20:24:47.426472 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.426446    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjv56\" (UniqueName: \"kubernetes.io/projected/b49e9a80-6655-44d3-bf4e-183a304589bd-kube-api-access-vjv56\") pod \"must-gather-jxd97\" (UID: \"b49e9a80-6655-44d3-bf4e-183a304589bd\") " pod="openshift-must-gather-rkv8j/must-gather-jxd97"
Oct 30 20:24:47.474415 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.474387    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5qnpg\"/\"openshift-service-ca.crt\""
Oct 30 20:24:47.474526 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.474511    6030 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5qnpg\"/\"kube-root-ca.crt\""
Oct 30 20:24:47.474914 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.474899    6030 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5qnpg\"/\"default-dockercfg-nds29\""
Oct 30 20:24:47.502336 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.502284    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d343c28e-bc12-4ad4-858f-516811d01ce4-must-gather-output\") pod \"must-gather-w769m\" (UID: \"d343c28e-bc12-4ad4-858f-516811d01ce4\") " pod="openshift-must-gather-5qnpg/must-gather-w769m"
Oct 30 20:24:47.502336 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.502330    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rbzn\" (UniqueName: \"kubernetes.io/projected/d343c28e-bc12-4ad4-858f-516811d01ce4-kube-api-access-7rbzn\") pod \"must-gather-w769m\" (UID: \"d343c28e-bc12-4ad4-858f-516811d01ce4\") " pod="openshift-must-gather-5qnpg/must-gather-w769m"
Oct 30 20:24:47.502466 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.502356    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t667\" (UniqueName: \"kubernetes.io/projected/735ce979-15c1-43fc-a42a-9d5bf7504446-kube-api-access-9t667\") pod \"must-gather-wnx25\" (UID: \"735ce979-15c1-43fc-a42a-9d5bf7504446\") " pod="openshift-must-gather-4jlph/must-gather-wnx25"
Oct 30 20:24:47.502466 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.502378    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/735ce979-15c1-43fc-a42a-9d5bf7504446-must-gather-output\") pod \"must-gather-wnx25\" (UID: \"735ce979-15c1-43fc-a42a-9d5bf7504446\") " pod="openshift-must-gather-4jlph/must-gather-wnx25"
Oct 30 20:24:47.502665 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.502648    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d343c28e-bc12-4ad4-858f-516811d01ce4-must-gather-output\") pod \"must-gather-w769m\" (UID: \"d343c28e-bc12-4ad4-858f-516811d01ce4\") " pod="openshift-must-gather-5qnpg/must-gather-w769m"
Oct 30 20:24:47.502709 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.502661    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/735ce979-15c1-43fc-a42a-9d5bf7504446-must-gather-output\") pod \"must-gather-wnx25\" (UID: \"735ce979-15c1-43fc-a42a-9d5bf7504446\") " pod="openshift-must-gather-4jlph/must-gather-wnx25"
Oct 30 20:24:47.515769 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.515747    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t667\" (UniqueName: \"kubernetes.io/projected/735ce979-15c1-43fc-a42a-9d5bf7504446-kube-api-access-9t667\") pod \"must-gather-wnx25\" (UID: \"735ce979-15c1-43fc-a42a-9d5bf7504446\") " pod="openshift-must-gather-4jlph/must-gather-wnx25"
Oct 30 20:24:47.526656 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.526634    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rbzn\" (UniqueName: \"kubernetes.io/projected/d343c28e-bc12-4ad4-858f-516811d01ce4-kube-api-access-7rbzn\") pod \"must-gather-w769m\" (UID: \"d343c28e-bc12-4ad4-858f-516811d01ce4\") " pod="openshift-must-gather-5qnpg/must-gather-w769m"
Oct 30 20:24:47.579620 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.579596    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rkv8j/must-gather-jxd97"
Oct 30 20:24:47.675925 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.675895    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jlph/must-gather-wnx25"
Oct 30 20:24:47.707634 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.707602    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5qnpg/must-gather-w769m"
Oct 30 20:24:47.851315 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.851288    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rkv8j/must-gather-jxd97"]
Oct 30 20:24:47.853812 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:24:47.853786    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb49e9a80_6655_44d3_bf4e_183a304589bd.slice/crio-dd1678ed54b3328ef0721c932bf100b622bf445a5ab19a057368f131e717f174 WatchSource:0}: Error finding container dd1678ed54b3328ef0721c932bf100b622bf445a5ab19a057368f131e717f174: Status 404 returned error can't find the container with id dd1678ed54b3328ef0721c932bf100b622bf445a5ab19a057368f131e717f174
Oct 30 20:24:47.855603 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.855588    6030 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:24:47.968018 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.967902    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jlph/must-gather-wnx25"]
Oct 30 20:24:47.970617 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:24:47.970586    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod735ce979_15c1_43fc_a42a_9d5bf7504446.slice/crio-4968d4870fe6de7de2b6ecc5838d0caf5810637b83b15eb7e46dd1c40f17c374 WatchSource:0}: Error finding container 4968d4870fe6de7de2b6ecc5838d0caf5810637b83b15eb7e46dd1c40f17c374: Status 404 returned error can't find the container with id 4968d4870fe6de7de2b6ecc5838d0caf5810637b83b15eb7e46dd1c40f17c374
Oct 30 20:24:47.990156 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:47.990134    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5qnpg/must-gather-w769m"]
Oct 30 20:24:47.992278 ip-10-0-142-167 kubenswrapper[6030]: W1030 20:24:47.992253    6030 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd343c28e_bc12_4ad4_858f_516811d01ce4.slice/crio-2ceed00d7fb040f0d4a3e07aa38169679789f3e580c95652ee4f99b8ab232200 WatchSource:0}: Error finding container 2ceed00d7fb040f0d4a3e07aa38169679789f3e580c95652ee4f99b8ab232200: Status 404 returned error can't find the container with id 2ceed00d7fb040f0d4a3e07aa38169679789f3e580c95652ee4f99b8ab232200
Oct 30 20:24:48.182652 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:48.182615    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rkv8j/must-gather-jxd97" event={"ID":"b49e9a80-6655-44d3-bf4e-183a304589bd","Type":"ContainerStarted","Data":"dd1678ed54b3328ef0721c932bf100b622bf445a5ab19a057368f131e717f174"}
Oct 30 20:24:48.183693 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:48.183661    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5qnpg/must-gather-w769m" event={"ID":"d343c28e-bc12-4ad4-858f-516811d01ce4","Type":"ContainerStarted","Data":"2ceed00d7fb040f0d4a3e07aa38169679789f3e580c95652ee4f99b8ab232200"}
Oct 30 20:24:48.184686 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:48.184662    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jlph/must-gather-wnx25" event={"ID":"735ce979-15c1-43fc-a42a-9d5bf7504446","Type":"ContainerStarted","Data":"4968d4870fe6de7de2b6ecc5838d0caf5810637b83b15eb7e46dd1c40f17c374"}
Oct 30 20:24:49.192539 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:49.191817    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jlph/must-gather-wnx25" event={"ID":"735ce979-15c1-43fc-a42a-9d5bf7504446","Type":"ContainerStarted","Data":"c298f42700e03f992d02dc0c9b2ed92f1b38ece12d7977dfcc457bd27be99825"}
Oct 30 20:24:49.192539 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:49.191875    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jlph/must-gather-wnx25" event={"ID":"735ce979-15c1-43fc-a42a-9d5bf7504446","Type":"ContainerStarted","Data":"d93b7e688a36970b3ca2b801451ccd34d7649cd0099e67597c68fb8d0f8abd33"}
Oct 30 20:24:49.215354 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:49.215306    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-4jlph/must-gather-wnx25" podStartSLOduration=1.452609501 podStartE2EDuration="2.215291938s" totalImagesPullingTime="762.682437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:49.214505628 +0000 UTC m=+3168.130112486" watchObservedRunningTime="2026-10-30 20:24:49.215291938 +0000 UTC m=+3168.130898793"
Oct 30 20:24:51.909327 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:51.909287    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6vm8t_b7a73dcc-6a78-4344-b0b3-7a8611466a41/global-pull-secret-syncer/0.log"
Oct 30 20:24:52.325167 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:52.325127    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qnmn2_0fef01e5-9be4-4368-8f1e-3f73d7d6c60d/konnectivity-agent/0.log"
Oct 30 20:24:52.445411 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:24:52.445329    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-167.us-east-2.compute.internal_2bdf808689747c42465325897cee67ca/haproxy/0.log"
Oct 30 20:25:00.360380 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:00.356188    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mgclc_7d40f8de-854b-4a46-bf24-a37b49e75583/manager/0.log"
Oct 30 20:25:01.131147 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:01.131109    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6pvgl_7f8b5c02-97b9-4018-ab0a-43af4476f135/manager/0.log"
Oct 30 20:25:02.588865 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:02.587634    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_96d833e9-2afc-45f9-a0aa-3c06cb5795b1/alertmanager/0.log"
Oct 30 20:25:02.616415 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:02.616384    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_96d833e9-2afc-45f9-a0aa-3c06cb5795b1/config-reloader/0.log"
Oct 30 20:25:02.659774 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:02.659706    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_96d833e9-2afc-45f9-a0aa-3c06cb5795b1/kube-rbac-proxy-web/0.log"
Oct 30 20:25:02.697385 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:02.697356    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_96d833e9-2afc-45f9-a0aa-3c06cb5795b1/kube-rbac-proxy/0.log"
Oct 30 20:25:02.737644 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:02.737602    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_96d833e9-2afc-45f9-a0aa-3c06cb5795b1/kube-rbac-proxy-metric/0.log"
Oct 30 20:25:02.798737 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:02.797752    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_96d833e9-2afc-45f9-a0aa-3c06cb5795b1/prom-label-proxy/0.log"
Oct 30 20:25:02.835071 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:02.830748    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_96d833e9-2afc-45f9-a0aa-3c06cb5795b1/init-config-reloader/0.log"
Oct 30 20:25:02.945073 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:02.944901    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cfmcp_4201a324-4bf9-46fc-b1be-c8381f3e9dba/kube-state-metrics/0.log"
Oct 30 20:25:02.990301 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:02.990266    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cfmcp_4201a324-4bf9-46fc-b1be-c8381f3e9dba/kube-rbac-proxy-main/0.log"
Oct 30 20:25:03.059600 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.059568    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cfmcp_4201a324-4bf9-46fc-b1be-c8381f3e9dba/kube-rbac-proxy-self/0.log"
Oct 30 20:25:03.214110 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.213595    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4jpzb_92d09c10-51ac-4231-8195-d0c853887162/node-exporter/0.log"
Oct 30 20:25:03.242066 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.241567    6030 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"]
Oct 30 20:25:03.258250 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.256978    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4jpzb_92d09c10-51ac-4231-8195-d0c853887162/kube-rbac-proxy/0.log"
Oct 30 20:25:03.296066 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.292117    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4jpzb_92d09c10-51ac-4231-8195-d0c853887162/init-textfile/0.log"
Oct 30 20:25:03.364716 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.364675    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"]
Oct 30 20:25:03.364938 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.364781    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:03.454300 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.454178    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/49a8a131-d20e-4131-a1b7-38eccde88506-podres\") pod \"perf-node-gather-daemonset-7hqcl\" (UID: \"49a8a131-d20e-4131-a1b7-38eccde88506\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:03.454300 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.454221    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49a8a131-d20e-4131-a1b7-38eccde88506-lib-modules\") pod \"perf-node-gather-daemonset-7hqcl\" (UID: \"49a8a131-d20e-4131-a1b7-38eccde88506\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:03.454300 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.454262    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49a8a131-d20e-4131-a1b7-38eccde88506-sys\") pod \"perf-node-gather-daemonset-7hqcl\" (UID: \"49a8a131-d20e-4131-a1b7-38eccde88506\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:03.454300 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.454293    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/49a8a131-d20e-4131-a1b7-38eccde88506-proc\") pod \"perf-node-gather-daemonset-7hqcl\" (UID: \"49a8a131-d20e-4131-a1b7-38eccde88506\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:03.454707 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.454329    6030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz8q7\" (UniqueName: \"kubernetes.io/projected/49a8a131-d20e-4131-a1b7-38eccde88506-kube-api-access-gz8q7\") pod \"perf-node-gather-daemonset-7hqcl\" (UID: \"49a8a131-d20e-4131-a1b7-38eccde88506\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:03.555940 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.555710    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz8q7\" (UniqueName: \"kubernetes.io/projected/49a8a131-d20e-4131-a1b7-38eccde88506-kube-api-access-gz8q7\") pod \"perf-node-gather-daemonset-7hqcl\" (UID: \"49a8a131-d20e-4131-a1b7-38eccde88506\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:03.555940 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.555788    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/49a8a131-d20e-4131-a1b7-38eccde88506-podres\") pod \"perf-node-gather-daemonset-7hqcl\" (UID: \"49a8a131-d20e-4131-a1b7-38eccde88506\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:03.555940 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.555824    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49a8a131-d20e-4131-a1b7-38eccde88506-lib-modules\") pod \"perf-node-gather-daemonset-7hqcl\" (UID: \"49a8a131-d20e-4131-a1b7-38eccde88506\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:03.556687 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.556636    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/49a8a131-d20e-4131-a1b7-38eccde88506-podres\") pod \"perf-node-gather-daemonset-7hqcl\" (UID: \"49a8a131-d20e-4131-a1b7-38eccde88506\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:03.556902 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.556881    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49a8a131-d20e-4131-a1b7-38eccde88506-sys\") pod \"perf-node-gather-daemonset-7hqcl\" (UID: \"49a8a131-d20e-4131-a1b7-38eccde88506\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:03.556986 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.556947    6030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/49a8a131-d20e-4131-a1b7-38eccde88506-proc\") pod \"perf-node-gather-daemonset-7hqcl\" (UID: \"49a8a131-d20e-4131-a1b7-38eccde88506\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:03.557089 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.557073    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/49a8a131-d20e-4131-a1b7-38eccde88506-proc\") pod \"perf-node-gather-daemonset-7hqcl\" (UID: \"49a8a131-d20e-4131-a1b7-38eccde88506\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:03.557162 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.557138    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49a8a131-d20e-4131-a1b7-38eccde88506-lib-modules\") pod \"perf-node-gather-daemonset-7hqcl\" (UID: \"49a8a131-d20e-4131-a1b7-38eccde88506\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:03.557229 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.557193    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49a8a131-d20e-4131-a1b7-38eccde88506-sys\") pod \"perf-node-gather-daemonset-7hqcl\" (UID: \"49a8a131-d20e-4131-a1b7-38eccde88506\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:03.572546 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.572520    6030 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz8q7\" (UniqueName: \"kubernetes.io/projected/49a8a131-d20e-4131-a1b7-38eccde88506-kube-api-access-gz8q7\") pod \"perf-node-gather-daemonset-7hqcl\" (UID: \"49a8a131-d20e-4131-a1b7-38eccde88506\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:03.754266 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.754234    6030 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:03.970552 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:03.970519    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-wwrm5_f6b01178-0262-47a5-81bb-d715f5cf4380/prometheus-operator/0.log"
Oct 30 20:25:04.004481 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:04.004419    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-wwrm5_f6b01178-0262-47a5-81bb-d715f5cf4380/kube-rbac-proxy/0.log"
Oct 30 20:25:04.094227 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:04.094193    6030 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"]
Oct 30 20:25:04.319710 ip-10-0-142-167 kubenswrapper[6030]: E1030 20:25:04.317693    6030 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xbxhg" containerName="extractor"
Oct 30 20:25:04.405638 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:04.405277    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl" event={"ID":"49a8a131-d20e-4131-a1b7-38eccde88506","Type":"ContainerStarted","Data":"ae2712b7b115b3b634b1f1df708f78a81b1af787532f0dbbd509786a2fd04592"}
Oct 30 20:25:04.405638 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:04.405351    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl" event={"ID":"49a8a131-d20e-4131-a1b7-38eccde88506","Type":"ContainerStarted","Data":"c35d44ac098c337370d5855e11d32dd74117e952ed3eebca4dcb6ee6dc3bf764"}
Oct 30 20:25:04.406114 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:04.406095    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:06.442823 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:06.442783    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5qnpg/must-gather-w769m" event={"ID":"d343c28e-bc12-4ad4-858f-516811d01ce4","Type":"ContainerStarted","Data":"c4c3ff8b0327245cf02b7f992ba14fee9f36138b016eb4aef144f99ef67501d3"}
Oct 30 20:25:06.443411 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:06.442835    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5qnpg/must-gather-w769m" event={"ID":"d343c28e-bc12-4ad4-858f-516811d01ce4","Type":"ContainerStarted","Data":"ebaa00b38904fb34a24973a314c238f0498138def7ab57d5d1c520aaca2ae3cd"}
Oct 30 20:25:06.445913 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:06.445877    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rkv8j/must-gather-jxd97" event={"ID":"b49e9a80-6655-44d3-bf4e-183a304589bd","Type":"ContainerStarted","Data":"48e02d9177946f86d95d764d8af37dc60deb6b839ddd38b40e4ad4d9521d9b9c"}
Oct 30 20:25:06.446100 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:06.445924    6030 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rkv8j/must-gather-jxd97" event={"ID":"b49e9a80-6655-44d3-bf4e-183a304589bd","Type":"ContainerStarted","Data":"65cbcc0c50ce49d76cc6e2b0cf6fbf65108b5ae899b2a2ed618d3913da2aac93"}
Oct 30 20:25:06.465754 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:06.465694    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl" podStartSLOduration=3.465676583 podStartE2EDuration="3.465676583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:04.434183911 +0000 UTC m=+3183.349790768" watchObservedRunningTime="2026-10-30 20:25:06.465676583 +0000 UTC m=+3185.381283440"
Oct 30 20:25:06.495788 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:06.495722    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5qnpg/must-gather-w769m" podStartSLOduration=1.588363 podStartE2EDuration="19.495699957s" totalImagesPullingTime="17.907336957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:06.466296133 +0000 UTC m=+3185.381902986" watchObservedRunningTime="2026-10-30 20:25:06.495699957 +0000 UTC m=+3185.411306815"
Oct 30 20:25:06.497095 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:06.496203    6030 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rkv8j/must-gather-jxd97" podStartSLOduration=1.615800332 podStartE2EDuration="19.496191567s" totalImagesPullingTime="17.880391235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:06.495436821 +0000 UTC m=+3185.411043678" watchObservedRunningTime="2026-10-30 20:25:06.496191567 +0000 UTC m=+3185.411798426"
Oct 30 20:25:07.727635 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:07.727603    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-grxfc_39be0ba6-63fe-4329-bfbd-7eed3d36be55/manager/0.log"
Oct 30 20:25:07.945076 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:07.943404    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-grxfc_39be0ba6-63fe-4329-bfbd-7eed3d36be55/manager/0.log"
Oct 30 20:25:08.773076 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:08.773021    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l7rcc_2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc/manager/0.log"
Oct 30 20:25:08.806070 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:08.806012    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cf964748-d8zhr_c2b7322b-ec51-4436-bedb-b4a923ffdf75/manager/0.log"
Oct 30 20:25:09.284193 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:09.284027    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l7rcc_2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc/manager/0.log"
Oct 30 20:25:09.320881 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:09.320780    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cf964748-d8zhr_c2b7322b-ec51-4436-bedb-b4a923ffdf75/manager/0.log"
Oct 30 20:25:09.491631 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:09.491586    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cf964748-d8zhr_c2b7322b-ec51-4436-bedb-b4a923ffdf75/manager/1.log"
Oct 30 20:25:09.855431 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:09.855382    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cf964748-d8zhr_c2b7322b-ec51-4436-bedb-b4a923ffdf75/manager/1.log"
Oct 30 20:25:10.722010 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:10.721949    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-89xm5_6a9b909a-d885-4ff2-8032-8079ac535f5a/proxy/0.log"
Oct 30 20:25:10.992343 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:10.992267    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-89xm5_6a9b909a-d885-4ff2-8032-8079ac535f5a/proxy/0.log"
Oct 30 20:25:11.180358 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:11.180320    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pzxdj_a675f8c3-6d56-485d-8c87-438d66ebdc1e/dns/0.log"
Oct 30 20:25:11.220173 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:11.220142    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pzxdj_a675f8c3-6d56-485d-8c87-438d66ebdc1e/kube-rbac-proxy/0.log"
Oct 30 20:25:11.328935 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:11.328801    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dwfmh_ef93f664-f7a4-42b9-bb9c-a10c519e9685/dns-node-resolver/0.log"
Oct 30 20:25:11.461901 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:11.461869    6030 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-7hqcl"
Oct 30 20:25:12.154500 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:12.154361    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-grxfc_39be0ba6-63fe-4329-bfbd-7eed3d36be55/manager/0.log"
Oct 30 20:25:12.468878 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:12.468792    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-grxfc_39be0ba6-63fe-4329-bfbd-7eed3d36be55/manager/0.log"
Oct 30 20:25:12.543528 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:12.543494    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bxxz2_35333d53-b864-432d-af6d-b1baec362af6/node-ca/0.log"
Oct 30 20:25:13.161077 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:13.157748    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l7rcc_2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc/manager/0.log"
Oct 30 20:25:13.228282 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:13.228249    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cf964748-d8zhr_c2b7322b-ec51-4436-bedb-b4a923ffdf75/manager/0.log"
Oct 30 20:25:13.800628 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:13.800539    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l7rcc_2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc/manager/0.log"
Oct 30 20:25:13.875181 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:13.874946    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cf964748-d8zhr_c2b7322b-ec51-4436-bedb-b4a923ffdf75/manager/0.log"
Oct 30 20:25:13.996672 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:13.996638    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cf964748-d8zhr_c2b7322b-ec51-4436-bedb-b4a923ffdf75/manager/1.log"
Oct 30 20:25:14.295925 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:14.295879    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cf964748-d8zhr_c2b7322b-ec51-4436-bedb-b4a923ffdf75/manager/1.log"
Oct 30 20:25:15.292614 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:15.292583    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-89xm5_6a9b909a-d885-4ff2-8032-8079ac535f5a/proxy/0.log"
Oct 30 20:25:15.588265 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:15.588190    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-89xm5_6a9b909a-d885-4ff2-8032-8079ac535f5a/proxy/0.log"
Oct 30 20:25:16.825257 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:16.825203    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-grxfc_39be0ba6-63fe-4329-bfbd-7eed3d36be55/manager/0.log"
Oct 30 20:25:16.976130 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:16.976086    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-grxfc_39be0ba6-63fe-4329-bfbd-7eed3d36be55/manager/0.log"
Oct 30 20:25:17.868637 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:17.868605    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l7rcc_2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc/manager/0.log"
Oct 30 20:25:17.918820 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:17.918766    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cf964748-d8zhr_c2b7322b-ec51-4436-bedb-b4a923ffdf75/manager/0.log"
Oct 30 20:25:18.133688 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:18.133655    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b99845f6f-rqs6w_2896aa38-c98c-4fa1-a8dd-3149d45e93b6/kube-auth-proxy/0.log"
Oct 30 20:25:18.284020 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:18.283989    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-clbw6_948623f8-d637-44f2-a460-43737e3384d1/istio-proxy/0.log"
Oct 30 20:25:18.342537 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:18.342489    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l7rcc_2a3f1f39-378f-460c-8b61-b8bfa8f2c2cc/manager/0.log"
Oct 30 20:25:18.359661 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:18.359628    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-x6hbg_db939b9a-f127-428e-bc8f-c137346b089c/istio-proxy/0.log"
Oct 30 20:25:18.389089 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:18.388991    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cf964748-d8zhr_c2b7322b-ec51-4436-bedb-b4a923ffdf75/manager/0.log"
Oct 30 20:25:18.543895 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:18.543865    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cf964748-d8zhr_c2b7322b-ec51-4436-bedb-b4a923ffdf75/manager/1.log"
Oct 30 20:25:18.891076 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:18.891017    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cf964748-d8zhr_c2b7322b-ec51-4436-bedb-b4a923ffdf75/manager/1.log"
Oct 30 20:25:19.856309 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:19.856269    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-89xm5_6a9b909a-d885-4ff2-8032-8079ac535f5a/proxy/0.log"
Oct 30 20:25:20.026069 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:20.024146    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b99845f6f-rqs6w_2896aa38-c98c-4fa1-a8dd-3149d45e93b6/kube-auth-proxy/0.log"
Oct 30 20:25:20.114120 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:20.113835    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-clbw6_948623f8-d637-44f2-a460-43737e3384d1/istio-proxy/0.log"
Oct 30 20:25:20.148500 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:20.148467    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-89xm5_6a9b909a-d885-4ff2-8032-8079ac535f5a/proxy/0.log"
Oct 30 20:25:20.153598 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:20.153573    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-x6hbg_db939b9a-f127-428e-bc8f-c137346b089c/istio-proxy/0.log"
Oct 30 20:25:21.809188 ip-10-0-142-167 kubenswrapper[6030]: I1030 20:25:21.809152    6030 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-crfs6_cfda46fc-34fd-461d-a2bf-d4ec19596ed2/serve-healthcheck-canary/0.log"